RWAPP

My name's Rob. I'm an iOS software development engineer for Capital One UK. I talk & write about mobile accessibility.

What the European Accessibility Act (Might) Mean for Mobile Development

The European Accessibility Act, or EAA is due to become law in Europe later this year, and it defines some specific requirements for mobile. In fact, its the first accessibility legislation that I’m aware of, anywhere, that explicitly covers mobile apps.


Since 2012 the European Union has been working on standardising accessibility legislation across Europe. The ultimate aim is to both improve the experience for those who need to use assistive technology, but also to simplify the rules business need to follow on accessibility. The years of discussions and consultations has lead to the European Accessibility Act, written in 2018, which covers a range of requirements for, amongst other channels, mobile.
While the EAA doesn’t blanket cover every category of app, the net is pretty broad. The act covers any apps sold or in use within the European Union that fall into these categories, so even if you’re not part of the Euro Zone if your app is available on an app store any country that is, the act applies to you. So, if you’re in the business of making mobile interactions, you’ll need to be prepared. Fortunately, both Apple and Android provide many of the tools required to conform to the law at a system level, meaning you’ll likely be a long way towards complying already.
As with any new law, until tested in the courts, it is somewhat open to interpretation. Also, I’m not a lawyer, I’m a mobile developer with a keen focus on accessibility, so this post makes up my personal thoughts about how I would try to comply with the law and will undoubtedly contain factual inaccuracies. I’ve not included everything but focussed on the areas I think are most useful to mobile developers. If you believe this new legislation may regulate your app, I’d highly recommend reading the act through yourself, and of course, you’ll need to get some legitimate legal advice.

Apps Types Covered

The EAA specifically covers mobile apps in certain areas, while this doesn’t cover all apps, it’s safe to say these categories make up a significant section of mobile apps available.

Transport

Any app related to publicly available transport including air, bus, rail and ‘waterborne’ transport. The act doesn’t specify taxis or ride sharing specifically, but arguably they could be included under the definition of ‘passenger transport services’

Banking

Any app that provides banking services — the language for this category uses the phrase ‘banking services’ specifically, suggesting the EU would apply this rule not only to banks themselves but more broadly to apps that use open banking to access other accounts or other banking related services.

e-commerce

Any app that allows for digitally purchasing either digital or physical goods or services. This clause will be the one that covers the majority of apps, and the requirements here are not as detailed as the more specific domains above.

Requirements for All Apps & Websites

Alternatives to Non-Text Content

If you present content in video form, make sure your video features subtitles or closed captioning, provide a transcript for any content presented in video or audio format. If you present text content in an image or as part of a video, add an accessibility label or subtitles that can be read by a screen reader.

Make Content Available to Screen Readers

This requirement covers two points from the act. Firstly:

Information content shall be available in text formats that can be used to generate alternative assistive formats to be presented in different ways by the users and via more than one sensory channel.

Also:

consistently … [present content] in a way which facilitates interoperability with a variety of user agents and assistive technologies.

My understanding of these paragraphs is that any text-based content should be accessible to screen readers (VoiceOver and TalkBack). Screen reader users make up by far the most significant constituency of assistive technology users. Also, both OSes also use the same techniques they use to support screen readers to allow support for other assistive technologies, so get screen readers right, and you’ll find other assistive technology will work well too.
For the most part, both Android & iOS built-in screen readers will do a great job of making the text available for the user, even making guesses at text included in images where needed. However, at times it can be easy to cause unexpected behaviour from screen readers — from elements being out of order to elements being missed entirely or read when not present on the screen. The best way to see if you need to make any changes here is to test with your devices screen reader enabled.

Requirements for Transport & Banking Apps & Websites

Flexible Magnification

It is unclear from the act whether magnification applies to simple screen magnification or to allowing magnification of text sizes. I suspect it would be relatively simple to argue that screen magnification fulfils this requirement if you’re looking to satisfy the minimum needed by the law. Screen magnification, however, isn’t a great experience for your customer, as it cuts context and reduces discoverability, so I’d highly recommend supporting dynamic text sizes regardless of the true meaning of the law.
Screen magnification is a system feature available on both platforms (Android, iOS) that requires no developer changes. Dynamic text sizes, however, need a little more consideration. It’s worth giving some UX thought to how a screen will look and function when the screen appears with the largest text, you will at times need to make decisions on compromises to design with accessibility text sizes.
iOS Developers will need to adopt the use of iOS’ built-in text styles and use the adjustsFontForContentSizeCategory property for each content label.
For Android, you should use SP sizes for text that scale from your standard size to match your customer’s device settings.

Flexible Contrast

In regular use, your text to background contrast ratio should be 4.5:1 for most text. With this setting enabled, you should look to have a contrast ratio of 7:1, so this might require some UX decisions as to what colours to use.
iOS provides an accessibility setting to increase contrast, as a developer, you can listen to UIAccessibility.isDarkerSystemColorsEnabled to see if you need to make UI changes based on this setting. I’d also highly recommend listening to UIAccessibility.isReduceTransparencyEnabled if you’re making use of transparency or blur in your app, and if the user has this setting enabled, be sure to provide an alternative. The alternative doesn’t have to be a solid colour, a reduced alpha or increased blur may be enough. A technique Apple make use of on SpringBoard is to use a solid colour that has a tint of the colour underneath.

Springboard
iOS SpringBoard with standard settings

Springboard
iOS SpringBoard with reduced transparency enabled
Android does provide a High Contrast Text accessibility setting, but unfortunately provides no developer documentation regarding what this does, or how developers can leverage it, so this may require a setting within your app to increase contrast.

Alternative Colour

Alternative colour is potentially the caveat that may need the most work to conform to; this probably needs someone more skilled than me in reading legislation to precisely know what this clause means.
It’s possible that the customer’s device’s built-in colour filters will fulfil this requirement, as inverting colours (Android, iOS), switching to greyscale, or adding a filter (Android, iOS) to the screen would have this effect. However, it could mean that you are expected to allow a setting for your customers to change your main body text colour. I’m not aware of a system framework on either platform to allow for simple switching of body text colours in this way, so may require some global skinning work for your app, and possibly some UX decisions.
The text says only ‘provide for an alternative colour to convey information,’ I’ll let you draw your own conclusion as to which option would allow your app to fulfil the requirement.

Alternative to Fine Motor Control

Both iOS and Android support control by external switch devices, or by using the device’s screen or buttons as a switch. Providing your app works as expected with screen readers, you’ll probably find switch control works fine, but it’s worth testing with this enabled.

Be More Like Neil

Toby Jones (as Neil) and Neil Baldwin
Toby Jones (as Neil) and Neil Baldwin

Like many, perhaps even all, of us in software engineering, and I’m sure other disciplines as well, I suffer from imposter syndrome. A crushing sense that I don’t, and never will, have the skills and knowledge to match my colleagues. I live in constant fear of being found out, that one day someone will come up to my desk and tell me they’ve figured out that I’ve been lying about my abilities all along, and that I need to pack my things and leave the building immediately. Its a feeling that caused me to take almost 10 years from writing my first code to even considering applying for a job in software, and when I first got a call from the recruiter, I genuinely thought she’d made a mistake and picked the wrong CV pile.

In reality, software is a field so incredibly vast and complex that no-one could ever know or understand everything, or even most things. We’re all learning and improving our skills all the time, and we all know different things as a result of our different experiences.

This weekend I was honoured to have the chance to meet a man who has been an inspiration to me for a few years now. A man for whom feeling he doesn’t belong has never caused him to question his abilities, perhaps quite the opposite - causing him to make every opportunity he could to do the things that made him happy. Someone who’s probably well known to you if you’re a 1990’s English league footballer, clown, or Church of England Bishop - a man named Neil Baldwin. I’m not the only person to be inspired by Neil, his friend, England’s best ever World Cup goal scorer Gary Lineker said of Neil “you want to be like him”. I think being more like Neil is a noble aim, and one we could probably all benefit from.

“I saw the Queen last week” Neil told me when I met him, in the same way you might mention to a friend you’d seen an old mutual acquaintance. I later found out he hadn’t just seen the Queen, but had been invited to Buckingham Palace to be presented with the British Empire Medal by Her Majesty, a medal he had pinned to his chest when we chatted, along with a small, seemingly random, selection of charity and football league lapel pins he had scattered on his jacket. This sums up Neil, he’s achieved a lot in his storied life, much of it is documented in the multi-BAFTA winning biopic Marvellous. Much of the rest you’ll find in his autobiography, standing out amongst his co-writer’s other titles, almost exclusively biographies of British Prime Ministers. All of those achievements are important to Neil, and he’s rightly proud, but none of them more important than any other.

Neil accepting a BAFTA for Marvellous
Neil accepting a BAFTA for Marvellous

Neil left school at 16, he was diagnosed with learning difficulties, and although he had no problem with school, it didn’t make him happy. Neil had made a simple decision, one we should probably all make, and more importantly remind ourselves of - that he wanted to be happy. Given that credo, there was a clear next step for Neil’s career - he joined the circus to become a clown. After a few years of falling of the back of a fire truck to make others laugh, Neil returned home to his family. Unemployed, Neil’s career goal was to manage Stoke City Football Club. So Neil made sure he was in the right place at the right time, and was invited to be Stoke City’s kit man. Stoke manager, Lou Macari who hired Neil directly described this as “the best signing I ever made”

Neil interrupting a TV interview with Stoke City manager Lou Macari
Neil interrupting a TV interview with Stoke City manager Lou Macari

Neil’s friends include Prince Edward - Neil just “knocked on his door” and introduced himself - the Archbishop of Canterbury - each one since the 60’s - and former England manager Kevin Keegan. He has an honorary degree from Keele University, and even played a match for his beloved Stoke City. Asked bout his learning difficulties, Neil replies “what difficulties?” Asked about what he’s achieved, Neil says: “I hope it shows people that you can do what you want with your life, if you keep at it.”

Neil accepting his degree from Keele University
Neil accepting his degree from Keele University

It’s natural to doubt yourself and your abilities, but there are a few lessons I think we can all take from Neil to help us remember we’re all capable of achieving the best we can.

  • Be happy
  • Make other people happy
  • Be confident in your own ability to do what makes you happy
  • Be proud of all your achievements, but don’t brag
  • Make your own opportunities

Perhaps being a little more like Neil might make us all happier.

Mars Watch Privacy Policy

Privacy Policy

Rob Whitaker built the Mars Watch app as an Open Source app. This SERVICE is provided by Rob Whitaker at no cost and is intended for use as is.

This page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.

If you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.

The terms used in this Privacy Policy have the same meanings as in our Terms and Conditions, which is accessible at Mars Watch unless otherwise defined in this Privacy Policy.

Information Collection and Use

For a better experience, while using our Service, I may require you to provide us with certain personally identifiable information, including but not limited to email address & name if you choose to contact me.

The app does use third party services that may collect information used to identify you.

Link to privacy policy of third party service providers used by the app

Log Data

I want to inform you that whenever you use my Service, in a case of an error in the app I collect data and information (through third party products) on your phone called Log Data. This Log Data may include information such as your device Internet Protocol (“IP”) address, device name, operating system version, the configuration of the app when utilizing my Service, the time and date of your use of the Service, and other statistics.

Cookies

Cookies are files with a small amount of data that are commonly used as anonymous unique identifiers. These are sent to your browser from the websites that you visit and are stored on your device’s internal memory.

This Service does not use these “cookies” explicitly. However, the app may use third party code and libraries that use “cookies” to collect information and improve their services. You have the option to either accept or refuse these cookies and know when a cookie is being sent to your device. If you choose to refuse our cookies, you may not be able to use some portions of this Service.

Service Providers

I may employ third-party companies and individuals due to the following reasons:

  • To facilitate our Service;
  • To provide the Service on our behalf;
  • To perform Service-related services; or
  • To assist us in analyzing how our Service is used.

I want to inform users of this Service that these third parties have access to your Personal Information. The reason is to perform the tasks assigned to them on our behalf. However, they are obligated not to disclose or use the information for any other purpose.

Security

I value your trust in providing us your Personal Information, thus we are striving to use commercially acceptable means of protecting it. But remember that no method of transmission over the internet, or method of electronic storage is 100% secure and reliable, and I cannot guarantee its absolute security.

Links to Other Sites

This Service may contain links to other sites. If you click on a third-party link, you will be directed to that site. Note that these external sites are not operated by me. Therefore, I strongly advise you to review the Privacy Policy of these websites. I have no control over and assume no responsibility for the content, privacy policies, or practices of any third-party sites or services.

Children’s Privacy

These Services do not address anyone under the age of 13. I do not knowingly collect personally identifiable information from children under 13. In the case I discover that a child under 13 has provided me with personal information, I immediately delete this from our servers. If you are a parent or guardian and you are aware that your child has provided us with personal information, please contact me so that I will be able to do necessary actions.

Changes to This Privacy Policy

I may update our Privacy Policy from time to time. Thus, you are advised to review this page periodically for any changes. I will notify you of any changes by posting the new Privacy Policy on this page. These changes are effective immediately after they are posted on this page.

Contact Us

If you have any questions or suggestions about my Privacy Policy, do not hesitate to contact me, rw@rwapp.co.uk.

This privacy policy page was created at privacypolicytemplate.net and modified/generated by App Privacy Policy Generator

Terms & Conditions

By downloading or using the app, these terms will automatically apply to you – you should make sure therefore that you read them carefully before using the app. You’re not allowed to copy, or modify the app, any part of the app, or our trademarks in any way. You’re not allowed to attempt to extract the source code of the app, and you also shouldn’t try to translate the app into other languages, or make derivative versions. The app itself, and all the trade marks, copyright, database rights and other intellectual property rights related to it, still belong to Rob Whitaker.

Rob Whitaker is committed to ensuring that the app is as useful and efficient as possible. For that reason, we reserve the right to make changes to the app or to charge for its services, at any time and for any reason. We will never charge you for the app or its services without making it very clear to you exactly what you’re paying for.

The Mars Watch app stores and processes personal data that you have provided to us, in order to provide my Service. It’s your responsibility to keep your phone and access to the app secure. We therefore recommend that you do not jailbreak or root your phone, which is the process of removing software restrictions and limitations imposed by the official operating system of your device. It could make your phone vulnerable to malware/viruses/malicious programs, compromise your phone’s security features and it could mean that the Mars Watch app won’t work properly or at all.

You should be aware that there are certain things that Rob Whitaker will not take responsibility for. Certain functions of the app will require the app to have an active internet connection. The connection can be Wi-Fi, or provided by your mobile network provider, but Rob Whitaker cannot take responsibility for the app not working at full functionality if you don’t have access to Wi-Fi, and you don’t have any of your data allowance left.

If you’re using the app outside of an area with Wi-Fi, you should remember that your terms of the agreement with your mobile network provider will still apply. As a result, you may be charged by your mobile provider for the cost of data for the duration of the connection while accessing the app, or other third party charges. In using the app, you’re accepting responsibility for any such charges, including roaming data charges if you use the app outside of your home territory (i.e. region or country) without turning off data roaming. If you are not the bill payer for the device on which you’re using the app, please be aware that we assume that you have received permission from the bill payer for using the app.

Along the same lines, Rob Whitaker cannot always take responsibility for the way you use the app i.e. You need to make sure that your device stays charged – if it runs out of battery and you can’t turn it on to avail the Service, Rob Whitaker cannot accept responsibility.

With respect to Rob Whitaker’s responsibility for your use of the app, when you’re using the app, it’s important to bear in mind that although we endeavour to ensure that it is updated and correct at all times, we do rely on third parties to provide information to us so that we can make it available to you. Rob Whitaker accepts no liability for any loss, direct or indirect, you experience as a result of relying wholly on this functionality of the app.

At some point, we may wish to update the app. The app is currently available on iOS – the requirements for system(and for any additional systems we decide to extend the availability of the app to) may change, and you’ll need to download the updates if you want to keep using the app. Rob Whitaker does not promise that it will always update the app so that it is relevant to you and/or works with the iOS version that you have installed on your device. However, you promise to always accept updates to the application when offered to you, We may also wish to stop providing the app, and may terminate use of it at any time without giving notice of termination to you. Unless we tell you otherwise, upon any termination, (a) the rights and licenses granted to you in these terms will end; (b) you must stop using the app, and (if needed) delete it from your device.

Changes to This Terms and Conditions

I may update our Terms and Conditions from time to time. Thus, you are advised to review this page periodically for any changes. I will notify you of any changes by posting the new Terms and Conditions on this page. These changes are effective immediately after they are posted on this page.

Contact Us

If you have any questions or suggestions about my Terms and Conditions, do not hesitate to contact me, rw@rwapp.co.uk.

This Terms and Conditions page was generated by App Privacy Policy Generator

Robot Pattern Testing for XCUITest

I recently spoke at iOSDevUK giving an overview of how we test our iOS app at Capital One. The most common follow up question I have been asked is regarding using the Robot Pattern.
I touched on the robot Pattern briefly in the slides in regards to how we make our end-to-end UI tests simple to follow, and have to admit I gave it little extra thought. But clearly this is something others are interested in knowing more about, and there’s very little about the Robot Pattern online, especially for iOS.

The Robot Pattern was designed by Jake Wharton at Square, when testing in Kotlin. As a result much of the information available focusses on Kotlin and Espresso testing. In fact, this is why we settled on the Robot Pattern, as it meant consistency in approach between our Android and iOS testing.
With that in mind, I’d highly recommend Jake’s talk on the Robot Pattern that offers a far better introduction than I will manage here. While Jake’s code may be Kotlin specific, the overall principals at the beginning of the talk apply to any UI.

Why Use the Robot Pattern

There’s three big reasons to use the Robot Pattern when writing XCUI Tests.

  1. Ease of understanding
    We came to XCUITests from Calabash, where our tests were written in Cucumber. Cucumber is close to natural language, meaning anyone, not just engineers fluent in that specific language could read and understand what the tests were testing, quickly and easily, without having to know exactly how the test works. Write the tests in native code, and there’s already a learning curve to knowing whats being tested and what isn’t.

  2. Reuse of code
    By breaking down our tests into steps, each implementation step can be re-used as many times as needed. If your app has a login screen before performing any action in the app, thats a lot of setup for each test. Instead you can just call login() each time, then move on to the more specific areas of your test. If you do need to do something a little different, you can pass parameters into the function.

  3. Isolating implementation details
    Whatever architecture your app uses, your aim is the single responsibility principle. Sticking to this allows you to switch out an object for a new one, with a new implementation, while still keeping the objects core functionality. This allows for easier to maintain, test, and improve code. So why should your tests be different? Jake Wharton describes this as separating the ‘What’ from the ‘How’. Your test should only be concerned with the ‘what’, meaning if your view changes how things appear or happen on screen, you don’t need to change your whole test suite.

Writing an XCUITest

Lets imagine we’re writing UI tests for Apple’s built in Messages app. If you load the app from suspended you’re greeted with a large title ‘Messages’ at the top of the screen, and a button to create a new message. Let’s assume we’re testing tapping on this button, and sending a new message to a user with iMessage. Our XCUITest might look something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
func test_sendNewiMessage() {
let app = XCUIApplication()
app.launch()

app.buttons["new_message"].tap()

let newMessage = app.staticTexts["New Message"]
let predicate = NSPredicate(format: "exists == true")
let expectation = XCTNSPredicateExpectation(predicate: predicate, object: newMessage)
let result = XCTWaiter.wait(for: [expectation], timeout: 5)
XCTAssertEqual(result, .completed)

app.typeText("iMessage Contact")

let newimessage = app.staticTexts["New iMessage"]
let newimessagePredicate = NSPredicate(format: "exists == true")
let newiMessageExpectation = XCTNSPredicateExpectation(predicate: newimessagePredicate, object: newimessage)
let newiMessageResult = XCTWaiter.wait(for: [newiMessageExpectation], timeout: 5)
XCTAssertEqual(newiMessageResult, .completed)

let firstField = app.textFields["messageField"]

firstField.typeText("test iMessage")
app.buttons["send"].tap()

let message = app.staticTexts["test iMessage"]
let messagePredicate = NSPredicate(format: "exists == true")
let messageExpectation = XCTNSPredicateExpectation(predicate: messagePredicate, object: message)
let messageResult = XCTWaiter.wait(for: [messageExpectation], timeout: 5)
XCTAssertEqual(messageResult, .completed)
}

It’s pretty clear there are several issues with this test - theres a lot of duplicated code and it’s difficult to follow what is being tested. But what if we also want to test sending a new message to an SMS contact, we’d have to duplicate this whole test. Then if we make a genuine change to the UI, we’ll have to change both tests.

Creating a Robot

During this test, we’re accessing two screens. The initial list of conversations, the one with the large heading ‘Messages’, then the conversation detail screen that appears once we tap the new message button. We’ll create a base Robot class that contains some common functions like asserting elements exist and tapping on the screen. Each screen will then have its own Robot class that extends Robot. These screen specific Robots contain actions specific to that screen, so our conversations list will contain one high level function to create a new message, our conversations detail has more for this test, as thats where we spend most of our time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import XCTest

class Robot {
var app = XCUIApplication()

func tap(_ element: XCUIElement, timeout: TimeInterval = 5) {
let expectation = XCTNSPredicateExpectation(predicate: NSPredicate(format: "isHittable == true"), object: element)
guard XCTWaiter.wait(for: [expectation], timeout: timeout) == .completed else {
XCTAssert(false, "Element \(element.label) not hittable")
}
}

func assertExists(_ elements: XCUIElement..., timeout: TimeInterval = 5) {
let expectation = XCTNSPredicateExpectation(predicate: NSPredicate(format: "exists == true"), object: elements)
guard XCTWaiter.wait(for: [expectation], timeout: timeout) == .completed else {
XCTAssert(false, "Element does not exist")
}
}
}
1
2
3
4
5
6
7
8
class ConversationListRobot: Robot {

lazy private var newConversationButton = app.buttons["new_message"]

func newConversation() -> ConversationDetailRobot {
tap(newConversationButton)
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
class ConversationDetailRobot: Robot {

private var messageType = "Message"
lazy private var screenTitle = app.staticTexts["New \(messageType)"]
lazy private var contactField = app.textFields["contact"]
lazy private var cancel = app.buttons["Cancel"]
lazy private var messageField = app.textFields["messageField"]
lazy private var sendButton = app.buttons["send"]

func checkScreen(messageType: String) -> Self {
self.messageType = messageType
assertExists(screenTitle, contactField, cancel, messageField, sendButton)
return self
}

func enterContact(contact: String) -> Self {
tap(contactField)
contactField.typeText(contact)
return self
}

func enterMessage(message: String) -> Self {
tap(messageField)
messageField.typeText(message)
return self
}

func sendMessage() -> Self {
tap(sendButton)
return self
}

@discardableResult
func checkConversationContains(message: String) -> Self {
let messageBubble = app.staticTexts[message]
assertExists(messageBubble)
return self
}
}

These then allow us to chain together the functions to create tests, so our example above becomes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func test_sendNewiMessage() {

let message = "test message"

XCUIApplication().launch()

ConversationListRobot()
.newConversation()
.checkConversationContains(message: "Message")
.enterContact(contact: "iMessage Contact")
.checkScreen(messageType: "iMessage")
.enterMessage(message: message)
.sendMessage()
.checkConversationContains(message: message)
}

Immediately this is easier to read and understand for anyone at a glance, without having to know how the app functions, or how to write Swift.

Building Blocks

These high-level functions can then be chained together in different configurations, depending on what we want to test. Let’s say we want to try the same, but with an SMS contact. We’d make a new test, changing our contact to an ‘SMS Contact’ and on our second checkScreen() our message type would be ‘Message’.
Using this technique we can easily build several tests built from the building blocks we’ve created, we can try invalid contacts, we can check what happens if the user goes back to the message list, then creates a new conversation, we could try not entering a message at all, and many more. All using these basic functions with different values passed or in a different order, or maybe adding a new one where needed. And if the button used to send the message changes, we only need to change this in the implementation of sendMessage(), not in every test.

I’d highly reccomend getting to grips with XCUITesting, its an incredibly simple way to test your app presents to your users as you’re expecting, and so far as I can tell is criminally underused. The Robot Pattern has proved a clean, simple technique that has solved many issues for us, so its really worth considering if it will do the same for you. There’s very little documentation on using the Robot Pattern with Swift and iOS, but Faruk Toptaş provides some sample Android Kotlin code on his writeup.

Feel free to drop me an email if you have any questions about using this technique, rw@rwapp.co.uk

New Japan Calendar

I’m a big fan of New Japan Pro Wrestling, New Japan offer a great streaming service called New Japan World that provides most of their events live, and the rest as VOD. Not being located within Japan Standard Time timezone, this sometimes means some early mornings. Unfortunatley NJPW events are not at regular times and days, starting any time from 6am to 11am. Meaning I mostly forget when the next event is. At times following NJPW can be a bit of a chore for a non-japaneese speaker.

So I’ve created a calendar for all upcoming NJPW evtents that you can subscribe to in your calendar app

The calendar is created using a web scraper that reads the details of the most recent events available at njpw1972.com
I hope you find the calendar useful in keeping up with New Japan, and I’ll post the source code soon.

Mars Watch Command Line

Inspired by by perennial iOS hacker Steve T-S‘s quest to make Lights Off available for every platform possible. I created a Swift command line app that runs Mars Watch in the Mac terminal.

If you’d like to compile yourself, source code is included with the rest of the Mars Watch repo. If you’re after the binary thats on GitHub too (direct download - if using Safari you’ll need to remove the extension and chmod +x it, I could figure out a better way to deliver this, but…)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
marswatch -h

usage:

Time Zones:
-m for Mars Coordinated Time
-o for time at Opportunity location
-c for time at Curiosity location
Multiple time zones valid
Displays Mars Coordinated Time if no option given

-n for continuous usage

-a About

I recommend full-screening terminal and running -a

WKWebview Challenges

Way back since iOS 8, Apple have been recommending we switch web content in our app to use the newer, in many ways more powerful, WKWebview, and away from the OG web view, UIWebView. Its not unreasonable to assume UIWebView will be deprecated at some point in the future, so if you have a legacy app that is still using the older technology, its worth taking the time to migrate. WKWebView contains some great features over UIWebView - its runs as its own separate process, meaning you have all of the system’s optimisations for fetching and rendering web content. Theres also awesome new tools for evaluating javascript on the loaded page, meaning you can read elements in your code, be notified if a property changes, and run javascript commands to modify the loaded page. If this all sounds like something you’d like to take advantage of I recommend NSHipster‘s introduction to the changes in WKWebview.

But from my own recent experience in migrating to WKWebView, the new technology still doesn’t quite feel finished. Its obvious Apple have made some trade-off decisions to make WKWebView‘s performance as good as it can be, but this sometimes impacts what control we have over it as a developer. I wanted to share some of the challenges I faced, as I’ve not seen a ton of discussion of these. Hopefully you’ll be reading this and know where I’m going wrong, but if not, hopefully my pitfalls will help you avoid the same.

WKWebView can’t be used in storyboards

Anecdotally it appears adoption of WKWebview is not as high as you might expect when Apple is heavily hinting the alternative will be deprecated. My guess at why this might be is that the WKWebview can’t be used in storyboards in apps targeting anything lower than iOS 11. As we’re not too far away from iOS 12 I suspect this won’t be an issue too longer, as those of us supporting iOS x - 1 won’t have this restriction. You can still drag a Web View onto your storyboard, but Apple’s fix for this issue is to add a compile time warning that you’re using this prior to iOS 11.

Instead this has to be created programatically

1
2
3
4
let webConfiguration = WKWebViewConfiguration()
webView = WKWebView(frame: .zero, configuration: webConfiguration)
view.addSubview(webView)
// obvs add some constraints

WKWebView crashes if a user taps on an image

The heading is perhaps a little unfair, this is completely expected behaviour, it just might not be anticipated. As with any webpage on iOS, the user can long-press an image to bring up a share sheet, one of these share options is to save the image to the camera roll. This requires user permission. So if you don’t set NSPhotoLibraryAddUsageDescription in your info.plist you risk users accidentally crashing your app by hovering on an image just a little too long.

No access to the URLRequest body

This is where things get a little more specific to my use case. WKNavigationDelegate includes some helpful callbacks - in fact, thats one of the reasons I was looking to migrate. webView(_ webView: WKWebView, didFail navigation: WKNavigation!, withError error: Error) allows your app to respond to errors in navigation - a 500 error returned by the server for example, that can’t be detected in UIWebView. However, as WKWebView is running entirely in its own process, theres little access to what is happening with requests made. The pre-navigation callback webView(_ webView: WKWebView, decidePolicyFor navigationAction: WKNavigationAction, decisionHandler: @escaping (WKNavigationActionPolicy) -> Void) returns a WKNavigationAction object, which includes the URLRequest. Except the request.httpBody is always nil. This means its not possible to determine navigation behaviour based on a form response. While there is a suggestion for getting body values on StackOverflow using the evaluateJavaScript command, this is not suitable for using on web pages you don’t control, and can’t guarantee there won’t be breaking changes, or on secure web pages where injecting javascript is not a great security decision.

Can’t intercept requests with NSRLProtocol

As the URLRequest is made out of process, its not possible to intercept this as you usually might intercept all other requests out of your app. This means its not possible to add custom headers, or in my case, mock the requests for testing. There are a couple of workarounds for this, neither are especially pretty.
Using webView(_ webView: WKWebView, decidePolicyFor navigationAction: WKNavigationAction, decisionHandler: @escaping (WKNavigationActionPolicy) -> Void) you can call cancel the navigation, add your custom header to the request, and make a new request. This creates another call to …decidepolicyfor…, so just make sure you have a check in there to prevent an infinite loop.

1
2
3
4
5
6
7
8
9
10
11
12
func webView(_ webView: WKWebView, decidePolicyFor navigationAction: WKNavigationAction, decisionHandler: @escaping (WKNavigationActionPolicy) -> Void) {

var request = navigationAction.request

if request.value(forHTTPHeaderField: "myCustomHeaderFeid") != nil {
decisionHandler(.allow)
return
}

decisionHandler(.cancel)
request.addValue("myCustomHeaderValue", forHTTPHeaderField: "myCustomHeaderFeid") webView.load(request)
}

My requirement for this was to mock the response the web view is receiving, so for obvious reasons I don’t want to make changes to production code to facilitate mocking. The route I took was to use an Apple private API. As this is code that will never be included in the production app, there is no risk in this being flagged by Apple at app review. Including a category extension in a file only added to the testing target allows us to route URLRequests through the usual delegate in your app.

1
2
3
4
5
6
7
8
9
10
extension URLProtocol {
class func registerWebView(scheme: String!) {
if let contextController: AnyObject = NSClassFromString("WKBrowsingContextController") {
if contextController.responds(to: NSSelectorFromString("registerSchemeForCustomProtocol:")) {
let _ = contextController.perform(NSSelectorFromString("registerSchemeForCustomProtocol:"), with: scheme)
}
}
}
}
}

Then register http & https by calling

1
2
URLProtocol.registerWebView(scheme: "http") 
URLProtocol.registerWebView(scheme: "https")

Update 27/6/18

Apple WebKit engineer Brady Edison tweeted

So I asked him about the issue I’ve not been able to find a suitible workaround for, getting access to the POST request body.

I’d be delighted to find more concrete ways to solve these shortcomings, the extra power WKWebView has over UIWebView makes it a great alternative. But as it stands, for some use cases its still not really there, even with 3 yeras of refinement from Apple. So if you do work out a better solution, please let me know - rw@rwapp.co.uk

Dog Rater Support

Thanks for using Dog Rater, I hope its been an invaluable tool for all your dog rating needs. I’m really pupset if you’ve had a problem, hopefully this page will help.

My phone is getting pretty hot

Dog Rater uses some pretty advanced machine learning and computer vision techniques to bring you the best ratings available, this does mean your device is doing a lot of work. We’ve worked hard to make sure this process is as efficient as possible and we’ll continue to improve. For now we recommend avoiding prolonged dog rating sessions.

The app rated a bad dog

The app only rates good dogs.

The app rated something that isn’t a dog

The app only rates good dogs.

Got a question not listed here?

Please feel free to email me, rw@rwapp.co.uk

If you enjoy the free Dog Rater app, please consider sending me some beer money.

Tutorial - Vision and Core ML - Live Object Detection in iOS 11

While WWDC 2017 may have not been one of the most announcement-packed WWDC’s ever, there’s no doubt over the features Apple is most proud of and sees the biggest opportunities for the future. Augmented Reality using ARKit, Machine Learning with Core ML, and pairing these with Vision and Metal for AR and VR. My initial thought, I’m sure the same as many other viewers, was that this technology is for would be reserved for the experts, and fans of really hard sums. But Apple’s implementation is mind-blowingly simple, and can really be added into almost any app in just a few minutes.

In this tutorial we’ll build Core ML and Vision Framework app for detecting objects in real time using your phone’s camera, similar to the one demoed in WWDC session 506. To begin you’ll need the Xcode 9 Beta, and you’ll need the iOS 11 Beta installed on your testing iPhone or iPad, meaning for now this tutorial is just for registered Apple Developers. But if you’re not one right now, this will all be freely available in the autumn. If you are registered, head over to Apple’s Developer Downloads page, and grab what you need. Bare in mind that this tutorial is written against iOS 11 Beta 2, so things could change a little by the final release.

This tutorial assumes some basic iOS Swift programming knowledge already, but I’ve tried to keep it as beginner friendly as possible. Please drop me an email to rw@rwapp.co.uk if there’s anything unclear.

Setting up the project

For this demo we’re going to do the object detection live, showing the camera, and overlay text on the screen telling us what Core ML can see. To begin, create a new project in Xcode 9 Beta and choose a Single View Application. Call your project Object Detector. Head to ViewController.swift and add a couple of @IBOutlets. One for a UIView we’re calling cameraView - this will be your ‘viewfinder’. Add a second @IBOutlet for a UILabel called textLayer - this will be our output of what Core ML has detected.

Open the Main Storyboard and drag on a blank view. Resize it to fit the screen and pin it to the edges. Now drag a label on to the view. Make sure this is topmost on the root view, not on top of the view you just added, or it will be obscured by the camera output. Position the label to the top layout guides and fill the guides left & right. Delete the text, set the text colour to something bright so you’ll see it over the image, and change the label to be 6 lines long. Finally add some suitable constraints to the label. Join up the IBOutlets you made earlier to the view and label.

To use the camera, we need to ask permission first, so head to info.plist and add a key for ‘Privacy - Camera Usage Description’ the string value can be anything you like for now, this is the message shown when asking permission for camera use at first launch.

Displaying the camera output

Now, to the code. First we need to use Vision to create an AVCApture session, this provides us with the live camera view, and also feeds the data from the camera into Core ML. Import AVFoundation and Vision. Head back to ViewController.swift. We need to set up a video capture delegate, so add AVCaptureVideoDataOutputSampleBufferDelegate to your class definition. Add the following code between your IBOutlets and viewDidLoad.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// set up the handler for the captured images
private let visionSequenceHandler = VNSequenceRequestHandler()

// set up the camera preview layer
private lazy var cameraLayer: AVCaptureVideoPreviewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)

// set up the capture session
private lazy var captureSession: AVCaptureSession = {
let session = AVCaptureSession()
session.sessionPreset = AVCaptureSession.Preset.photo


guard
// set up the rear camera as the device to capture images from
let backCamera = AVCaptureDevice.default(.builtInWideAngleCamera,
for: .video,
position: .back),
let input = try? AVCaptureDeviceInput(device: backCamera)
else {
print("no camera is available.");
return session

}
// add the rear camera as the capture device
session.addInput(input)
return session
}()

The code above sets up the camera preview layer, then defines the capture session, telling iOS which camera to use, and what quality we’re looking for.

Now we’ve set that up, we need to add the camera preview layer to the display, and tell iOS to start capturing from the camera. Add the following code to viewDidLoad.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// add the camera preview
self.cameraView?.layer.addSublayer(self.cameraLayer)

// set up the delegate to handle the images to be fed to Core ML
let videoOutput = AVCaptureVideoDataOutput()

// we want to process the image buffer and ML off the main thread
videoOutput.setSampleBufferDelegate(self,
queue: DispatchQueue(label: "DispatchQueue"))

self.captureSession.addOutput(videoOutput)

// make the camera output fill the screen
cameraLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill

// begin the session
self.captureSession.startRunning()

Then to make sure the viewfinder fills the screen, add a viewDidLayoutSubviews override.

1
2
3
4
5
6
override func viewDidLayoutSubviews() {
super.viewDidLayoutSubviews()

// make sure the layer is the correct size
self.cameraLayer.frame = (self.cameraView?.frame)!
}

Set your target to run on your testing device (as this uses the camera, you’ll just get a blank screen on the simulator) and hit Build and Run. After granting access to the camera, you’ll be greeted with a full screen output of your devices rear camera view.

Missing something? Get the code up to this point on Github.

Adding Core ML Magic

First of all, we need a Machine Learning Model. Fortunately that’s a simple as downloading one from Apple’s website. Head over there now and download the Inception v3 model. Add this file to your project.

We now need to get the data from the AVCaptureSession, and feed that our Inception v3 Model. We do this using a AVCaptureVideoDataOutputSampleBufferDelegate method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
func captureOutput(_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection) {

// Get the pixel buffer from the capture session
guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

// load the Core ML model
guard let visionModel:VNCoreMLModel = try? VNCoreMLModel(for: Inceptionv3().model) else { return }

// set up the classification request
let classificationRequest = VNCoreMLRequest(model: visionModel,
completionHandler: handleClassification)

// automatically resize the image from the pixel buffer to fit what the model needs
classificationRequest.imageCropAndScaleOption = VNImageCropAndScaleOptionCenterCrop

// perform the machine learning classification
do {
try self.visionSequenceHandler.perform([classificationRequest], on: pixelBuffer)
} catch {
print("Throws: \(error)")
}
}
}

Just before we handle the results Core ML gives us, two functions to help us update the UI, and to map the Core ML result into something more meaningful

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func updateClassificationLabel(labelString: String) {
// We processed the capture session and Core ML off the main thread, so the completion handler was called onthe the same thread
// So we need to remember to get the main thread again to update the UI

DispatchQueue.main.async {
self.textLayer?.text = labelString
}
}

func textForClassification(classification: VNClassificationObservation) -> String {
// Mapping the results from the VNClassificationObservation to a human readable string
let pc = Int(classification.confidence * 100)
return "\(classification.identifier)\nConfidence: \(pc)%"
}

Finally, to handle the Core ML results themselves. Here we want to make a few quick checks to make sure everything is as we expect, then we need to filter to get the accuracy and data that we want. For this example we just want to display all the objects we’ve found, so the only filtering we’ve done is to cut off the last 20% of results. In the real world, you’ll probably want to look for specific objects, so use .filter here to narrow those down. More on that below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func handleClassification(request: VNRequest,
error: Error?) {
guard let observations = request.results else {

// Nothing has been returned so we want to clear the label.
updateClassificationLabel(labelString: "")

return
}

let classifications = observations[0...3] //taking just the top 3, ignoring the rest
.flatMap({ $0 as? VNClassificationObservation}) // discard any erroneous results
.filter({ $0.confidence > 0.2 }) // discard anything with less than 20% accuracy.
.map(self.textForClassification) // get the text to display
// Filter further here if you're looking for specific objects

if (classifications.count > 0) {
// update the label to display what we found
updateClassificationLabel(labelString: "\(classifications.joined(separator: "\n"))")
} else {
// nothing matches our criteria, so clear the label
updateClassificationLabel(labelString: "")
}
}

We only filtered through the top 3 results provided by the Machine Learning model. For this project we just want to know the most probable results. The observation results are provided in decreasing order of certainty, so we know anything else is likely to be a false positive. Click on the model in the sidebar, and you’ll see in the model description Apple lists a ‘top-5 error’ rate. This is the percentage of false positives provided for the model in the top 5 results when tested. In real world situations using a live camera, this is likely to be higher.

Description: Detects the dominant objects present in an image
from a set of 1000 categories such as trees, animals, food, vehicles,
person etc. The top-5 error from the original publication is 5.6%.

Iterating through all the provided guesses from the model is going to use mote resources, and is unlikely to yield any more real positives. If you’re looking of specific items and you’re doing your own filtering, you may find better end results by looking at more of the model’s output.

Doggo

Where to go from here

If you are looking for specific objects, I’d suggest cutting the confidence level right down to 10% or even less. Then add an extra filter step with your own custom logic to narrow down what you’re looking for.
I found the Inception v3 model to sometimes be a little too specific at times, and sometimes not specific enough. It all depends on the category of object you’re looking for, and wether the model has already been trained on that category. Its worth checking out other models to make sure you get the one that works best for your project. Apple provide some popular models ready to drop in to your project. Just change the model name in the line guard let visionModel:VNCoreMLModel = try? VNCoreMLModel(for: Inceptionv3().model) else { return } to match the name of the model you choose. If you find another model that does the job, Apple provide Core ML Tools to convert machine learning models to the Core ML format.

I hope you found this quick introduction to Core ML and Vision helpful, if you’ve got anything to add, or any questions, please let me know by email rw@rwapp.co.uk, and I’d love to see any apps you make using these frameworks, so please drop me a link when its on the App Store.

Full tutorial code available on Github