Over the past several years the barrier to entry for adopting machine learning and computer vision-related features has dropped substantially. Still, many developers are...
WWDC 2018: Opening Day. Augmented Reality, Machine Learning, and Siri Shortcuts
WWDC 2018 began this week in San José, CA, with the usual excitement of thousands of developers from all over the world
that could not wait to learn what new shiny objects Apple would unveil for them. The expectant developer community was not
disappointed. Even though the announcements were largely evolutionary and no new hardware of any kind was unveiled,
the enhancements to all four platforms (iOS, macOS, watchOS, tvOS) and the development tools were many, and hint at big
things to come.
Here are some of the more significant announcements.
I have always appreciated that with most new releases of system software, Apple tries to retain as much compatibility with
older devices as possible, extending their usable life. The newly announced iOS 12, promises to support the same range of
devices as iOS 11 and the system is undergoing a serious performance review that will significantly benefit the older ones.
Now, when the system senses increased CPU demand it ramps-up clock speed faster, leading to more responsive devices and an earlier
ramp-down that preserves batteries. Figures of 50 to 70% performance improvements were mentioned.
Several apps will undergo redesigns and enhancements in iOS 12, such as News, Stocks, Voice Memos, and iBooks. One of the more
interesting updates is Face Time, which will now include Group Face Time with up to 32 participants.
Group sessions use a vertical stack of video tiles that automatically move and resize to bring the current speaker into focus.
Conveniently, a Messages text chat group can easily become a Group Face Time.
The Photos app will boast vastly improved searching based on image content, effects suggestions also based on content, and
easier sharing. If you share with a friend a photo from an event you both attended, the recipient’s device will search for
related pictures from the same event and offer to share them back with you, so that you both can have all available pictures.
Your wish has finally come true. iOS 12 will allow you to create an animoji version of yourself, called a “Memoji”. The camera
looks at you and makes an animoji version of you that is highly customizable. You can edit your hair style, eye color,
add glasses and hats. And the best part, if you stick your tongue out, the updated vision framework will recognize it and add
it to your memoji. The memoji can be shared with your friends with Messages, or you can use it instead of your real self in
Do Not Disturb, Notifications, Screen Time
Apple has recognized the concern many of us have that we are spending too much time with our iOS devices. iOS 12 improves the Do Not Disturb functionality by hiding notifications on the home screen to remove all temptation to look at them.
Notifications can now be grouped rather than listed individually, with settings per app to control the grouping behavior. Grouped notifications
will allow you to quickly triage classes of notifications as you search for the most important ones.
Finally, Screen Time will produce reports you can study to determine how you are spending time with your phone. Are you playing
that game too much or spending too much time with that social networking app? You will be able to set time limits for those attention
hogs and get a dismissible block screen when you reach that time limit. Through iCloud Family Sharing, you will also be able to set time
allowances for your children.
The WWDC Keynote announced the next version of macOS. macOS 10.14 will move away from the California mountains, and into the desert,
as “macOS Mojave”.
Mojave will include many smaller improvements to the Desktop and Finder:
- Desktop Stacks. The stacks we have been using on the dock are breaking loose and will help you organize those loose files on your desktop.
- Gallery View. A new view mode for Finder windows that displays a large preview for a file.
- Enhanced Quick Look Quick Actions allow you to view and edit a file without launching an app.
- Easier and more flexible screen shot functionality.
- Dynamic Desktop will allow you to cycle through different desktop appearances during the day.
More notable improvements include the following.
You are probably familiar with the current Continuity feature that allows you to seamlessly transfer tasks from your phone to your computer,
such as editing an email or reading from a web site. Continuity camera will let you get a picture from your phone, as you work from
your computer. Say that you are preparing a presentation in Keynote and you need to insert a picture. You can activate your phone
camera from Keynote, take a picture, and it will be easily inserted into your document, without any kind of tedious file exchange process.
Perhaps the most hyped new feature of macOS Mojave is Dark Mode. It will let you change the appearance of your computer to a, well,
darker mode. It does look gorgeous and if, as a developer, you chose to participate, you can allow your app to switch modes as well.
Dark Mode is opt-in for your application. If you chose to activate the capability, your app will probably not look great right away and
you will have to do some work. If you mostly use standard controls, you will not have much to do. Asset catalogs will help you as
they now include dark mode placeholders for image assets. If you supply these extra images, they will be correctly displayed as the user
switches modes. Custom controls will require your attention to change their tint depending on the current mode.
The announced version 10 of Xcode makes good use of Dark Mode with a very attractive dark interface.
Some of the biggest announcements presented at the 2018 WWDC include more than one Apple platform.
This is perhaps one of the more exciting new features for developers. The machine learning gears within your iOS and macOS devices
will allow Siri to discover some patterns in your user behavior. Let’s say that you have a small morning ritual of opening some
applications and looking at some specific information. Siri will discover this pattern and offer to make a shortcut you can launch
in the future with a phrase of your choosing. After creating the shortcut, you will be able to start your morning by telling
Siri something like: “Start morning routine”. As a developer, you can do the minimum and simply allow Siri to include your app in a
shortcut, or you can actually expose specific parts of your application to Siri and handle more fine-grained shortcut requests.
This is the beginning of making your apps voice-controlled.
Privacy and Security
Recent events in the news have definitely increased our concerns about data security. Apple has responded in several ways:
- API protections for the use of application data, and resources such as the camera and microphone have been extended.
- Many web sites track a user by analyzing device configuration (fonts installed, plug-ins available, etc.). This “fingerprinting” technique is becoming more difficult in the new versions of Safari for iOS and macOS.
- Automatic passwords are more secure. The system can now detect repeated use of passwords and offer to replace them. Codes for two-factor authentication are now detected when sent to Messages and automatically added to the appropriate field.
A more important change for developers is “Notarized Apps”. This is an extension of the developer ID and certifies that apps are
trustworthy no matter how they are distributed to the user (e.g. a web-distributed Mac application). In the future Apple will require
that all macOS apps be notarized.
iOS Apps in macOS
Several new apps were announced for macOS Mojave, such as News, Stocks, Voice Memos and Home. This sounds pretty trivial
until you realize that these are all iOS apps. As it turns out, Apple has added parts of the UIKit framework into macOS so that you
can easily port iOS applications to macOS. This is very exciting but the details are still scarce, as the feature will be released
AR and USDZ file format
Apple is clearly excited about Augmented Reality. ARKit 2.0 has improvements in face tracking, environment textures, image
detection and tracking in 2 and 3D, recognition, and sharing of locations (maps).
An exciting new feature is the USDZ file format (Universal Scene Description). This file format allows Augmented Reality to be portable.
A USDZ file can, for example, be embeded in a web page so that an object can be examined through AR instead of a simple picture.
ML and CreateML
The current emphasis in Machine Learning is to make it compelling and easy to use by all developers. CoreML 2.0 is faster, better at
recognizing faces and face features, and can better detect people so that they can be removed from a picture and transferred to another.
CoreML also supports language recognition.
The new CreateML framework makes it easier to extend and train a model for image or language recognition. You can use Xcode and
Playgrounds to easily extend a model and train it by dragging and dropping data into the playground. The resulting model will only
contain the extension data and will be smaller and more convenient to bundle with your app.
Finally, Xcode is receving some well-deserved attention in version 10. Notable enhancements include:
- For macOS developers, NSGridView is now available in the Interface Builder library.
- An attractive Dark Mode.
- Better code completion.
- New refactoring tools for Swift.
- Better stability for code editing.
- Integration with source control to mark changes in code since the last checkout. The changes are marked with a color-coded bar on the gutter.
- Improved testing with inclusion and exclusion of tests, test order randomization, and parallel testing with different suites.
- Multi-Cursor Editing. You can now easily select and edit columns of code, or select multiple similar functions to refactor them in an equivalent way simultaneously.
There you have an overview of the most important announcements made during the first day of the 2018 WWDC. I hope many of these will
get you interested to get more details from the upcoming session videos. Also keep an eye out for more technically detailed blogs from
Big Nerd Ranch during the week.