<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://akyuu.aaaab3n.moe/feed.xml" rel="self" type="application/atom+xml" /><link href="https://akyuu.aaaab3n.moe/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-09-07T13:06:27-04:00</updated><id>https://akyuu.aaaab3n.moe/feed.xml</id><title type="html">求聞史紀</title><subtitle>What broke, how it was diagnosed, and the exact fix. Only things that really happened, with dates.</subtitle><entry><title type="html">IPv6 works, DHCP renews fine, ARP answers, and not one IPv4 packet gets through</title><link href="https://akyuu.aaaab3n.moe/2026/08/26/ipv6-fine-dhcp-renews-not-one-ipv4-packet-forwarded/" rel="alternate" type="text/html" title="IPv6 works, DHCP renews fine, ARP answers, and not one IPv4 packet gets through" /><published>2026-08-26T00:00:00-04:00</published><updated>2026-08-26T00:00:00-04:00</updated><id>https://akyuu.aaaab3n.moe/2026/08/26/ipv6-fine-dhcp-renews-not-one-ipv4-packet-forwarded</id><content type="html" xml:base="https://akyuu.aaaab3n.moe/2026/08/26/ipv6-fine-dhcp-renews-not-one-ipv4-packet-forwarded/"><![CDATA[<p>A two-minute link flap on the WAN port, and afterwards IPv4 was dead in both directions while IPv6 was
untouched. Every check on the router said healthy. The ISP gateway had quietly taken the public IP back
onto itself while still handing that same IP to the router over DHCP.</p>

<h2 id="symptom">Symptom</h2>

<p>The router is a MikroTik running RouterOS 7. The ISP box is an AT&amp;T BGW fiber gateway in
<strong>IP Passthrough</strong> mode: it leases its own public IPv4 to the router over DHCP, so the router holds the
public address directly. IPv6 does not go through passthrough; it is plain routing plus DHCPv6-PD.
Those two facts matter below.</p>

<p>On 2026-08-26 the WAN port flapped at 21:07:45 and came back at 21:09:00 (<code class="language-plaintext highlighter-rouge">link-downs</code> went from 4 to 6).
After that:</p>

<table>
  <thead>
    <tr>
      <th>Observation</th>
      <th>What it says</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>IPv6 egress completely normal, <code class="language-plaintext highlighter-rouge">traceroute</code> goes straight out</td>
      <td>Physical link and ISP are fine</td>
    </tr>
    <tr>
      <td>ARP for the IPv4 gateway is <code class="language-plaintext highlighter-rouge">complete</code> / <code class="language-plaintext highlighter-rouge">reachable</code></td>
      <td>Layer 2 is up, the BGW is answering ARP</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">/ip dhcp-client renew</code>, and <code class="language-plaintext highlighter-rouge">release</code> + <code class="language-plaintext highlighter-rouge">renew</code>, both <strong>succeed</strong>, same IP, lease refreshed</td>
      <td>DHCPv4 traffic passes, the BGW is still handing out the address</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">ping</code> to the gateway, to any public IPv4, <code class="language-plaintext highlighter-rouge">traceroute</code>: <strong>all time out</strong></td>
      <td>Not one IPv4 packet is forwarded outbound</td>
    </tr>
    <tr>
      <td>From an <strong>outside</strong> host, <code class="language-plaintext highlighter-rouge">ping</code> to the public IP <strong>works</strong>, but <code class="language-plaintext highlighter-rouge">curl :80</code> and <code class="language-plaintext highlighter-rouge">:443</code> fail</td>
      <td>Something is answering for that IP, and it is not us</td>
    </tr>
    <tr>
      <td>The <code class="language-plaintext highlighter-rouge">dstnat</code> port-forward rule counter: <strong>0 packets</strong></td>
      <td>Nothing inbound reaches the router either</td>
    </tr>
  </tbody>
</table>

<h2 id="why-it-is-easy-to-misdiagnose">Why it is easy to misdiagnose</h2>

<p>Every signal on the router itself looks healthy. The interface is up, ARP resolves, DHCP renews and the
lease is refreshed with the same address. That combination screams “router problem”, so the first hour
went into router-side actions: <code class="language-plaintext highlighter-rouge">renew</code>, <code class="language-plaintext highlighter-rouge">release</code> + <code class="language-plaintext highlighter-rouge">renew</code>, staring at routes and firewall rules.
None of it could help, and the lease alone explains why: the ISP hands out a <strong>14-week</strong> lease, so
RouterOS would never re-request on its own anyway.</p>

<p>The one signal that should have short-circuited all of this was the external probe: the public IP
answered <code class="language-plaintext highlighter-rouge">ping</code> from outside but refused TCP. The router was not answering that ping. The BGW was.</p>

<p>There was also a self-healing hook (<code class="language-plaintext highlighter-rouge">netwatch</code> calling a script) that was supposed to handle exactly this.
It had never fired once since it was created, for an unrelated reason: the script’s <code class="language-plaintext highlighter-rouge">policy</code> list
contained <code class="language-plaintext highlighter-rouge">policy</code>, and RouterOS silently refuses to run such a script from netwatch. Even fixed, it
could not have helped here, see the end of the Fix section.</p>

<h2 id="root-cause">Root cause</h2>

<p>After the flap the BGW re-established its own WAN session and kept the public IPv4 <strong>for itself</strong>, while
its passthrough side continued to lease that same IPv4 to the router. Both devices now believed they owned
the address. The BGW answered ARP and ICMP for it upstream and downstream, and forwarded nothing in either
direction. IPv6 was unaffected because it never depended on passthrough.</p>

<h2 id="proof">Proof</h2>

<p>Two checks, both from the router, decide it:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/ip firewall nat reset-counters
# then from a host outside this network: curl -m 5 http://&lt;public-ip&gt;/
/ip firewall nat print stats where comment~"Forward"
</code></pre></div></div>

<p>Counter stays at 0 packets: the inbound request never reached the router, so whoever answered the
external <code class="language-plaintext highlighter-rouge">ping</code> is in front of it. Combined with the successful DHCP renew, the only device that fits is
the BGW.</p>

<p>One trap on the way: <code class="language-plaintext highlighter-rouge">/ip firewall filter reset-counters</code> did <strong>not</strong> actually zero the counters on this
RouterOS build. <code class="language-plaintext highlighter-rouge">/ip firewall nat reset-counters</code> works. Use the NAT one for this test.</p>

<h2 id="fix">Fix</h2>

<p><strong>Emergency path first</strong>, because the real fix needs either the BGW’s web UI or a phone app.</p>

<p>The config already had a static default route via the BGW’s private LAN address, permanently inactive
because there was no on-link route to it. Adding a <code class="language-plaintext highlighter-rouge">/32</code> route is not enough: the source address would
still be the public IP, which the BGW ignores. What works is giving the WAN port an address in the BGW’s
own private subnet:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/ip address add address=192.168.3.2/24 interface=ether1 comment="BGW mgmt access"
</code></pre></div></div>

<p>Two things happen at once: the BGW becomes reachable (its web UI answers), and the static default route
flips to active, so outbound IPv4 comes back through the BGW’s ordinary NAT.</p>

<p><strong>But now there are two default routes at distance 1</strong>, the static one and the DHCP one, and RouterOS
load-balances across them (both show <code class="language-plaintext highlighter-rouge">+</code> in <code class="language-plaintext highlighter-rouge">/ip route print</code>). Half of the flows go into the black hole:
<code class="language-plaintext highlighter-rouge">8.8.8.8</code> works, <code class="language-plaintext highlighter-rouge">9.9.9.9</code> dies, the same destination succeeds and fails in turns. Move the DHCP route out
of the way in the same breath:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/ip dhcp-client set [find interface="ether1"] default-route-distance=10
/ip dhcp-client renew [find interface="ether1"]
</code></pre></div></div>

<p>This is double NAT. Every inbound port forward is dead while it is in place. It buys time, nothing more.</p>

<p><strong>The real fix</strong> is to make the BGW re-run passthrough, from easiest to hardest:</p>

<ol>
  <li>Restart the gateway from the ISP’s mobile app. This is what fixed it on 2026-08-26.</li>
  <li>BGW web UI: Restart, or toggle IP Passthrough off, save, on, save. Needs the emergency address above and the Device Access Code printed on the unit.</li>
  <li>Pull the power for about 30 seconds.</li>
</ol>

<p>After the restart: the public IPv4 was unchanged, the IPv6 prefix was unchanged (no LAN renumbering),
and every inbound port answered again from outside. The <code class="language-plaintext highlighter-rouge">/ip cloud</code> “Router is behind a NAT” warning
cleared after <code class="language-plaintext highlighter-rouge">force-update</code>.</p>

<p><strong>Then roll the emergency path back</strong>, otherwise the static route (distance 1) keeps winning over the
DHCP route (distance 10) and port forwards never return:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/ip address remove [find comment~"BGW mgmt access"]
/ip dhcp-client set [find interface="ether1"] default-route-distance=1
/ip dhcp-client renew [find interface="ether1"]
</code></pre></div></div>

<p>Remove the address <strong>first</strong>, then restore the distance. The other order opens a window with both routes
at distance 1, which is the ECMP black hole again.</p>

<h2 id="lessons">Lessons</h2>

<ul>
  <li><strong>When DHCP renews but nothing forwards, stop debugging the router.</strong> The upstream device is holding the address. Prove it from outside with a counter, not from inside with <code class="language-plaintext highlighter-rouge">ping</code>.</li>
  <li><strong>An emergency path is not done until its rollback is written next to it</strong>, including the order of operations. This one has two ways to hurt you: ECMP on the way in and a stuck static route on the way out.</li>
  <li><strong>Two default routes at equal distance is a half-working network</strong>, which is worse than a dead one because it looks like flakiness. Check <code class="language-plaintext highlighter-rouge">/ip route print</code> for two <code class="language-plaintext highlighter-rouge">+</code> entries before blaming anything else.</li>
  <li>Self-healing that “bounces DHCP” cannot fix a peer that is lying over DHCP. Design the fallback around <code class="language-plaintext highlighter-rouge">check-gateway=ping</code> and route distances, not around scripts.</li>
</ul>]]></content><author><name></name></author><summary type="html"><![CDATA[A two-minute link flap on the WAN port, and afterwards IPv4 was dead in both directions while IPv6 was untouched. Every check on the router said healthy. The ISP gateway had quietly taken the public IP back onto itself while still handing that same IP to the router over DHCP.]]></summary></entry></feed>