iPhone 4S, Exchange ActiveSync and internal wifi

It has been known for a while that iPhones and other iDevices do not play well with Exchange ActiveSync when roaming between a public network (such as 3G) and a private internal network to which the Exchange Server is connected. In particular, push email often does not work, which seems to be a bug in the iOS software. It’s a known issue, according to Apple. However, it caught me out recently, because the problem seemed to go away for a long time with the release of the iPhone 4.

At work, we have a set-up which is quite common for organisations of our size. We have two distinct networks: the internal network, which is reserved only for trusted devices owned and managed by us (the PCs, laptops, printers, switching gear, servers and now, thin clients). With 1000s of devices on this network, it is VLANed quite heavily to increase manageability, although I will admit this project was something I only completed fairly recently. Before last year, it was a single broadcast domain… but that is another story.

However, we also have a guest network. The guest network is isolated into its own VLAN, and is for wired clients which cannot authenticate as domain members (via 802.1x authentication) or for wireless clients connecting via a “Guest” SSID issued by our wireless controller. The guest network is still restricted to internal use – users authenticate to our RADIUS servers from their phones or laptops. Provided they provide valid credentials, they are provided with restricted access to the Internet.

All of the networks are linked together by our Forefront TMG deployment. This is driven by our inbound ISP connection and exposes several interfaces to the network – the internal network has two, teamed interfaces (for redundancy and throughput for data from cache) and the guest network has a further interface. The TMG deployment is the gateway for the guest network, and the internal network has a 0.0.0.0 default route for unresolved traffic crossing the VLANs.

When the Forefront TMG was provisioned last year, I initially configured the guest network both for internet access, but also with an internal set of “relay” rules, if you like, for access to certain resources on the internal network – OWA, RD Web Access, our management system, internal websites and, crucially, DNS lookups via our internal name servers. In effect, guest traffic was not NATed onto its own public IP. When it matched a firewall rule for one of our internal services, it was simply routed into the internal network. This made the deployment much simpler, and meant the internal IP addresses returned by internal DNS nameservers would still work for guest clients. Upshot: I don’t need more nameservers!

At the time, this did not pose a problem, even with the iPhone and iPad devices used by our staff. These phones could have been on 3G and wifi simultaneously, and we never had an issue with the mismatched IP addresses on the two networks stopping ActiveSync working.

That is, however, until someone upgraded to the iPhone 4S.

As noted in the blog post linked above:

“push” may stop working if your company’s Exchange ActiveSync server has a different IP address for intranet and Internet clients. Make sure the DNS for your network returns a single, externally-routable address to the Exchange ActiveSync server for both intranet and Internet clients

The problem experienced with this one iPhone 4S user went beyond push email. The user’s phone worked perfectly when away from the network. However, the moment it roaming onto our wifi, it seemed to have an adverse effect on the Exchange account configuration. Almost immediately, the phone would report a password error on a manual email check. The Exchange account would then refuse to work at all – on any network – until the user deleted the device from the Exchange Control Panel (ECP), switched back to 3G and re-created his connection.

I was not convinced the issue was with Exchange – all manner of other devices, even the iPhone 4, were still working. Nothing tested incorrectly. The problem was not a user issue, as I had him configure a test user account for a few days. Same problem.

Eventually, after a lot of painstaking troubleshooting (and waiting for feedback), it started to become very clear the issue was present only on the iPhone 4S and only in certain circumstances. However, it was much more serious than before – when the issue occurred, it did not just stop the iPhone from working until it roamed off-site again. It essentially wrote the email capability on the device off.

The resolution was a simple one, and one I should probably have implemented in the first place. The Forefront TMG deployment was re-configured. No routing was permitted between the internal network and the guest network. Instead, I added a network rule for guest network traffic to be NATed to its own public IP. I built a new cluster of standalone DNS servers, which serve two purposes – recursive lookups from the internal network (they are, effectively, caching servers) and hosting of the public DNS zones which return public IP addresses for all our network services.

When the guest network was given access to these nameservers, the iPhone 4S problem immediately went away. As detailed by Apple, it seems their devices are once again having issues with multiple IP addresses being issued by DNS for the same service. I thought this inconvenience had been resolved, but it would appear this design strategy will be going back into my network design methodology in the future. In any event, it did allow me to streamline and simplify our guest network configuration, which is always a good thing!

Watch out for Apple devices and the problem with issuing different internal and external IPs if they are used on your internal wifi. Either make the public IP routable internally and use that for internal access, or – a very common solution – don’t use them on wifi at all.

Exch 2010 SP1 with AirSync (iPhone/iPod/iPad)

Over the last couple of days, I took the time to upgrade my personal Exchange environment to Exchange 2010 SP1 Rollup 1 (I was on 2010 RTM). The update appeared to go without a hitch, but a day or so later, I discovered my iPod (in fact, this is true for any Apple iWhatever device) wouldn’t sync mail over the air via EAS, it wouldn’t send email and OWA replies/forwards failed with ugly error messages.

If you’re seeing any of the following errors just after upgrading to SP1, you might find the root cause and the associated fix is very simple – if so, read on.

• An error occurred while delivering this message
• This message has not been downloaded from the server
• Cannot Get Mail – the connection to the server failed
• In OWA: An unexpected error occurred and your request couldn’t be handled
• In event traces, imceaDomain must be a valid domain name.

Now, this issue was picked up and discussed in the release notes for 2010 SP1 as a known issue, but I didn’t clock this initially because I didn’t exactly read the notes thoroughly – a brief scan, perhaps, late in the evening, but nothing looked relevant on my tired eyes at the time. I’ve also already performed a number of SP1 upgrades elsewhere without experiencing issues, so I didn’t consider it important to refresh my memory by re-reading the notes.

The exact symptoms:

On the ActiveSync device:

Your email will sync, but there won’t be any content. The preview of the message text will display in the folder view, so you know something is there, but expanding the message to actually read it reveals the message: This message has not been downloaded from the server. Scrolling down, you can use the button to download remaining content – but it claims the message is 0 bytes in size and pressing this doesn’t do anything.

 

Message view - the messages just don't download

 

Sending email from the device resulted in a message: “Cannot Send Mail. An error occurred while delivering this message.” Unfortunately, all the errors issued by the Apple kit are fairly generic (probably because, in this instance, it didn’t actually know what the problem was – but I’m inclined to think it’ll always make you dig to find the root cause).

Non-Exchange accounts, such as Gmail, and potentially accounts on other Exchange environments configured differently to avoid this bug, worked absolutely fine.

Via OWA:

OWA, again, reports a generic error:

Expanding the details in the error or looking in your server’s event log, reveals an interesting exception message:

Exception message: imceaDomain must be a valid domain name.

If you’re not familiar with IMCEA (Internet Mail Connector Encapsulated Address), it was originally a method of inter-connecting mail environments by providing temporary addresses to users sending email via SMTP, but did not possess an SMTP email address. The mail system handled the encapsulation and subsequent reverse process in order to send and receive email for the user. The technology is still used today in the latest versions of Exchange, and you will often see cases where an SMTP address is unknown, so an IMCEA version of an X.500 address is displayed – often in NDR reports. According to Technet, Exchange actually uses IMCEA encapsulation for any address other than the default authoritative domain.

In this case, Exchange is having issues dealing with just that – the default authoritative domain. You see, if the friendly name you gave it has a space in it, or some other illegal character for a domain name, it triggers an error in the programming, which ultimately leads to this major loss of core messaging functionality.

As always, the fix is fairly simple. Remove any spaces from the friendly names of your accepted domains. You can do this at shell or the console – I prefer using the shell, in which case, use

Set-AcceptedDomain “Friendly Name of your default authoritative domain” -name “AnyNameWithoutSpacesOrIllegalCharacters”

Once the name change is complete, throw a restart on the MS Exchange AD Topology service during a period of planned system outage and functionality should come back. I pick that service because it restarts most of the others at the same time.

As a result of this issue, I will be forcing a new naming convention everywhere I manage Exchange, whereby accepted domains are ALWAYS named after the actual domain name for ALL accepted domains, thus containing no spaces and no other illegal characters. It transpired that the other sites were named in this fashion for the default domain anyway, which explains the reasoning as to why I never experienced this issue with those users.

Missing some cmdlets at Exchange Management Shell? Me too!

On one of our many Exchange Servers at work, I recently discovered the Exchange cmdlets in the Management Shell which I rely on for my daily Exchange management had disappeared. get-excommand reported just one Exchange cmdlet was loaded: Get-ExchangeDiagnosticInfo. Strange. They were there one day, gone the next. No, it wasn’t caused by an update to the best of my knowledge; it didn’t happen over our patching window.

The case of missing cmdlets was traced back to an issue with my user profile on this server. A test with another user account yielded no issues at the Management Shell.

A quick fix to this might be to obliterate the user profile using the System applet Control Panel, then log back in and have Windows generate a new profile. However, this is totally unnecessary and you’ll lose any special configuration, given how simple the actual solution is.

Exchange Management Shell uses a directory in the user’s roaming Application Data to store the Powershell module configuration settings. My module data had some… modifications. I don’t know the source of these changes, but it rendered the cmdlets missing. I suspected this was the case because shell loaded much more quickly than normal when it was broken – rather than show the status of the pending implicit remoting session, which I am used to seeing, it loaded and connected almost instantaneously.

The solution is to remove the C:\Users\username\AppData\Roaming\Microsoft\Exchange\RemotePowershell\your.domain.com directory.

After deleting this directory, restart the Shell. The startup process will create the directory and re-generate the module files, fixing your issue and allowing you to get on with whatever you needed to do!

Matt

—

P.s. I know I’ve been quiet lately, and for that, I apologise. For the past couple of months I’ve been involved in an almighty migration job, away from an awful managed service network (tip: NEVER opt for an outside company to supply your network. It falls apart!) to a vanilla Windows Server system. This came not a moment too soon but completing a migration of this magnitude for 2500 seats in the 6 week maintenance window is no easy feat!

I do have some articles on the backburner, and hope to get some out to you ASAP. Thanks for your patience, and thanks for reading!

Exchange 2010 SP1 Announced

Was it really 7 months ago Exchange 2010 RTMed? I find that incredibly hard to believe, but true. Today, the blogosphere heated up following the Exchange Team’s announcement of the first Service Pack for the latest and greatest version of Microsoft’s Exchange email server.

The team suggest Beta code will be available for your test environments later in the second quarter of 2010, sometime around June. Although I have no quibbles with Exchange 2010, the product group has still found places to make some useful improvements:

• Outlook Web Access (OWA) – performance improvements using more Web 2.0 AJAX-style programming, usability improvements and prettifying through the re-introduction of themes.
• Archiving and Compliance – My clients will benefit significantly from the ability to separate an archive mailbox from the user’s main mailbox – even to different mailbox stores. Frankly, server-side archive support was a long-awaited replacement to the problematic PST file or third-party tools, but moving data around within the same store just made no sense and we held off enabling the feature.

  Users with local Outlook AutoArchive PST files can also have their PST data imported directly into their Archive mailbox. With a bit of luck, we’ll begin to see many more PST free establishments.

• EMC/ECP Improvements – minor improvements are being made to the GUI management tools, negating the requirement for you to drop to Powershell for some configuration tasks.

Not mentioned were any bug fixes, although I suspect there will be a few. With a little hope, my pet hate – the preferred Global Catalog issue when creating and mounting a new Mailbox Database – will be resolved.

The Exchange Team have done a stellar job with the 2010 release so far and, unlike Exchange 2007, no real areas are lacking in functionality. However, there is always room for improvement and I look forward to seeing Beta code to play with later in the year!

See the full press release on the MS Exchange Team blog