ohoachuck / wwdc-downloader

WWDC 2019 video downloader script written in Swift - no external dependency.

/!\ Whatch out: current script does not work if you try to download directly to an external Hard Drvie !

WWDC Video sessions bulk download (wwdcDownloader.swift)

I went to the WWDC Debug lab and found out that current script does not work with shebang (#!/usr/bin/swift) due to a bug in Swift 5.1 (I will file a radar).

So the workaround (waiting for Swift 5.1 bug fix) is to compile the script before runing it (no crash with swiftc).

Synthax: wwdcDownloader.sh <same params as describes bellow>

example: wwdcDownloader.sh --hd720 --pdf --sample (will download all files in the same folder)

---

wwdcDownloader.swift script is a Swift script that should work out of the box without any needs of extra software or development package (at least ususaly :) ).

Its main purpose is to let you bulk download all WWDC session videos, pdf resources and sample codes in one shot.

Latest version is wwdcDownloader.swift.

Ok, this script is not the best in class solution for getting WWDC videos and other resources. There are multiple version of scripts that does the same out there. But the best in class reference is the nice designed mac application done by Guilherme Rambo : WWDC. You definitely want to check he's website.

The current scripts was mainly created to get in one shot all videos at the end of DubDubDC right before you run back home (in an external hard drive for instance). It's a good move to take advantage of WWDC conference center fast cable connection.

Using the options below, you can choose to retrieve 1080p, 720p or SD videos and request to download pdf and sample codes as well.

Note: script will download videos/pdfs in the current directory.

1080p videos

Downloading 1080p videos requires video processing. The script will attempt to use ffmpeg if available otherwise, will download the stream files but will not convert. The conversion process can be started after all videos are downloaded. After installing ffmpeg, re-running wwdcDownloader the same way as the first time will only convert the downloaded stream files to video files. You can install ffmpeg via [Homebrew (ffmpeg)] (https://formulae.brew.sh/formula/ffmpeg) (brew install ffmpeg) if you are downloading the 1080p videos.

Usage

./wwdcDownloader.swift

Downloads by default WWDC 2019 HD 1080p videos sessions (need ffmpeg).

Unless you plan to watch the videos on your TV you might just want to do get HD versions (720p):

./wwdcDownloader.swift --hd720 --pdf --sample

for taking HD videos, PDFs and sample codes when available.

Options

You can try wwdcDownloader.swift --help for more options.

Usage: wwdcDownloader.swift [--wwdc-year <year>] [--tech-talks] [--hd1080] [--hd720] [--sd] [--pdf] [--pdf-only] [--sample] [--sample-only] [--sessions <s1 s2 ...>] [--list-only] [--help]

Examples:

	- Download all 1080p videos for wwdc 2019 (default):
		./wwdcDownloader.swift --hd1080
		
	- Download all HD (720p) videos, slides PDF & the sample codes for wwdc 2019:
		./wwdcDownloader.swift --hd720 --pdf --sample

	- Download all 720p videos for wwdc 2019:
		./wwdcDownloader.swift --hd720
		
	- Download all SD videos for wwdc 2019:
		./wwdcDownloader.swift --sd
		
	- Download all SD videos & the slides PDF for wwdc 2019:
		./wwdcDownloader.swift --sd --pdf

	- Download all SD videos, slides PDF & the sample code for wwdc 2019:
		./wwdcDownloader.swift --sd --pdf --sample
	
	- Download only all PDF for wwdc 2019:
		./wwdcDownloader.swift --pdf-only
	
	- Download only all sample code for wwdc 2019:
		./wwdcDownloader.swift --sample-only

	- Download only SD videos + PDFs for sessions 503 and 504 for wwdc 2019:
		./wwdcDownloader.swift --sd --pdf --sessions 503 504

	- List titles of known sessions for wwdc 2019:
		./wwdcDownloader.swift --list-only

	- Download all 1080p videos for wwdc 2019:
		./wwdcDownloader.swift --wwdc-year 2019

Requirements

• Works on macOS.
• ffmpeg (for 1080 HD videos).

Related content

WWDC native app done by Guilherme Rambo. You definitely want to check he's website.

wwdc-downloader's Issues

Very inefficient and slow if downloading "directly" to external volume

First, thanks for the years of providing this tool. It's been very handy!

I commend you diving in to use Swift and Cocoa for this year's iteration. An unfortunate consuquence, I guess of the way that NSURLSession works, is when you run the tool on an external volume, the process is slowed down by the fact that NSURLSession downloads the file first to a temporary folder on the boot volume, and then the script copies the download to the destination folder. With each video comprising multiple GB of data, this causes a several minute delay on my Mac between the download of a video, and the beginning of downloading the next one.

It sure seems like an oversight that NSURLSession doesn't support specifying a volume on which the temporary download file should be saved.

I don't really expect a fix, just wanted to share my observation here in case somebody thinks of a clever/easy solution. I think that the "right" fix might unfortunately be moving away from NSURLSession's default downloader task

Additional video size other than SD and HD

Apple only provide two official video size for download. SD is 640p and HD is 720p. However, the embedded player will serve you up to 1080p if bandwidth allows. It's downloadable with a few extra steps and nothing difficult to download them.

I'm not sure if there are any interest in getting the higher resolution. If there is, I would consider working on adding into the scripts.

Please let me know.

Is there any ways to download unsynchronize?

I use your script to make an list about all the download urls.Maybe can use wget -i download-list.txt.
And in China, we can just copy the urls in Thunder,and can download many file at once.
wwdc hd url.txt
wwdc sd url.txt

Thanks!

Just want to say thank you for this application! Just works and does the job!

How large a download is the whole thing?

Hard coded home directory should be replace by a tilde

WWDC_DIRNAME is defined using an absolute path to the user directory.
While this is valid in most of the cases, it causes problems when a custom user directory has been defined.
/Users/${USER} should be replaced by '~', that will expand to the home directory defined in the user profile.

Thanks for that script, it was very handy !

No checking on partial download

I have hit an issue where a lot of the files (videos and sample code) did not complete download but the script will move the ".download" partial files to the final filename. Not sure if it is apple's side or my ISP being flakey but the issue remains that the script should check curl's exit status before moving the part file to the final filename.

The fix is simple and I have fixed it on my forked version. Can I make a pull request to include my fixes?

Some Sample Code not downloading properly

Most of the sample code downloads properly, but some don't.

For instance, 226 - Advanced NSOperations.zip only appears as a 2 kB download that is empty.

Ditto for 217 - AlignmentGuides - AppKit Haptic Feedback Filtering using NSAlignmentFeedbackFilter, although 217 - ForceTouchCatalog - Using the Force Touch Trackpad API downloads fine.

List of Videos

It's no way to see full list of video for previous years
Command
./wwdcVideoPDFGet-curlVersion.sh -y 2014 -L

starts downloading

Can not download WWDC2013 videos?

Hi , I want to download WWDC2013-405 video and url is https://developer.apple.com/videos/play/wwdc2013-405/,
and I try ./wwdcVideoPDFGet-curlVersion.sh -y 2013 -s 405 on iTerm2,
it return this :

Using 'no password' mode (this is possible since WWDC 2014 sessions addition => Apple bug ?)!
try using -l option if download does not work.

sed: /tmp/wwdc-session-8E67B4C3-3AD8-48C4-B949-0D1C51E189D0.tmp/titles.txt: No such file or directory
./wwdcVideoPDFGet-curlVersion.sh: line 46: /tmp/wwdc-session-8E67B4C3-3AD8-48C4-B949-0D1C51E189D0.tmp/titles.txt: No such file or directory
******* DOWNLOADING PDF FILES ********
******* DOWNLOADING SD VIDEOS ********

It just create a folder on my local disk.Then I type ./wwdcVideoPDFGet-curlVersion.sh -y 2013 -s 405 -l [email protected]

it turn out this:

Password:
sed: /tmp/wwdc-session-30B909D2-6D8E-481B-A83F-44E6EDC31178.tmp/titles.txt: No such file or directory
./wwdcVideoPDFGet-curlVersion.sh: line 46: /tmp/wwdc-session-30B909D2-6D8E-481B-A83F-44E6EDC31178.tmp/titles.txt: No such file or directory
******* DOWNLOADING PDF FILES ********
******* DOWNLOADING SD VIDEOS ********

How to solve this ?

Regex needs adjustment

I think there's been some changes to the session pages, I had to adjust the regex for it to detect any sessions:

from
let pat = "\\b.*\\/videos\\/play\\/wwdc2016\\/([0-9]*)\\/\"><h5\\b"
to
let pat = "\\b.*\\/videos\\/play\\/wwdc2016\\/([0-9]*)\\/\">.*"

Not sure if the regex could be better.

Does not work with Swift 3.1

I'm trying to compile and run wwdc2016.swift on OSX 10.12.4

Apple Swift version 3.1 (swiftlang-802.0.48 clang-802.0.38)
Target: x86_64-apple-macosx10.9

The code fails to compile.

errors.txt

Add HomeBrew integration

Consider to add HomeBrew integration in order to make it easy to install?

Keeps trying to download 2019 keynote? (and crashes)

OK so for some reason when I run the downloader script ./wwdcDownloader.swift it tries to download the WWDC 2019 Keynote video, which then fails, presumably because it isn't available. But I can't figure out why it's trying to do this.

For extra strange points I popped the code into an Xcode command line project and it works fine (downloads them to whatever DerivedData directory the thing outputs to). I notice when it fails from the command line it's trying to download a 1080p keynote video but in Xcode it's just downloading 720p everything.

In Xcode, where it succeeds

Downloading HD videos in current directory 
Let me ask Apple about currently available sessions. This can take some times (15 to 20 sec.) ...

[Session 101] : Keynote
Video : Video is not yet available !!!
...

On the command line, where it fails

Downloading 1080p videos in current directory
Let me ask Apple about currently available sessions. This can take some times (15 to 20 sec.) ...

[Session 101] : Keynote
Video : 101_1080p_keynote.mp4
[Session 101] Getting 101_1080p_keynote.mp4:
Stack dump:
0.    Program arguments: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift -frontend -interpret ./wwdcDownloader.swift -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -color-diagnostics -module-name wwdcDownloader 
0  swift                    0x00000001062cbee3 PrintStackTraceSignalHandler(void*) + 51
1  swift                    0x00000001062cb6bc SignalHandler(int) + 348
2  libsystem_platform.dylib 0x00007fff5e42eb5d _sigtramp + 29
3  libsystem_platform.dylib 0x00007f9c59f3c410 _sigtramp + 4222671056
4  libswiftCore.dylib       0x00007fff5dc66c09 _swift_updateClassMetadataImpl(swift::TargetClassMetadata<swift::InProcess>*, swift::ClassLayoutFlags, unsigned long, swift::TypeLayout const* const*, unsigned long*, bool) + 265
5  libswiftCore.dylib       0x00007fff5dc66c4b swift_updateClassMetadata2 + 27
6  libswiftCore.dylib       0x000000010b26212b swift_updateClassMetadata2 + 2908730619
7  libswiftCore.dylib       0x00007fff5dc6eb77 swift::MetadataCacheEntryBase<(anonymous namespace)::SingletonMetadataCacheEntry, int>::doInitialization(swift::ConcurrencyControl&, swift::MetadataCompletionQueueEntry*, swift::MetadataRequest) + 215
8  libswiftCore.dylib       0x00007fff5dc643f3 swift_getSingletonMetadata + 579
9  libswiftCore.dylib       0x000000010b26200f swift_getSingletonMetadata + 2908741215
10 libswiftCore.dylib       0x000000010b25746d swift_getSingletonMetadata + 2908697277
11 libswiftCore.dylib       0x000000010b25e90d swift_getSingletonMetadata + 2908727133
12 libswiftCore.dylib       0x000000010b25f599 swift_getSingletonMetadata + 2908730345
13 libswiftCore.dylib       0x000000010b25f5f8 swift_getSingletonMetadata + 2908730440
14 libswiftCore.dylib       0x00007fff5dae5004 $sSTsE7forEachyyy7ElementQzKXEKF + 452
15 libswiftCore.dylib       0x000000010b249b0f $sSTsE7forEachyyy7ElementQzKXEKF + 2910211279
16 swift                    0x0000000102b6338d llvm::MCJIT::runFunction(llvm::Function*, llvm::ArrayRef<llvm::GenericValue>) + 365
17 swift                    0x0000000102b69762 llvm::ExecutionEngine::runFunctionAsMain(llvm::Function*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, char const* const*) + 1090
18 swift                    0x0000000102132831 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 58929
19 swift                    0x000000010212092e swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 6862
20 swift                    0x00000001020be9ce main + 1246
21 libdyld.dylib            0x00007fff5e2433d5 start + 1
22 libdyld.dylib            0x000000000000000a start + 2715536438
Bus error: 10

This is with source I fetched just now with the PRs that just got merged for video streams and ffmpeg and so forth. Presumably this script works for folks so I'm not sure what I'm doing wrong.

HD PDFs aren't downloading correctly. Here's the fix.

I noticed that when choosing HD, the PDFs were not downloading correctly. Looks like around line 650 the "pdfURL" variable is getting set twice with the SD version winning out. But there's no "_sd_" pattern to remove in the HD version. As a result, the PDF link generated is invalid.

You can replace those two "pdfURL" commands with this one:

pdfURL=`echo ${videoURL} | sed 's/_'${LC_FORMAT}'_/_/g' | sed 's/\.mp4/\.pdf/g'`

Download Progress

i would like to have something like

Downloading xxx/xxx [50%] sample code for session 303: StoreKitSuite

Then you'll know how long it will take to download it all.

Add ability to download WWDC2011/WWDC2012 videos

Is there way to add ability to download WWDC2010/WWDC2011/WWDC2012 videos? They were available also.

https://developer.apple.com/videos/wwdc/2010/
https://developer.apple.com/videos/wwdc/2011/
https://developer.apple.com/videos/wwdc/2012/

Videos don't have audio

I downloaded every WWDC 2019 videos to an external HDD before abandoning unlimited, high speed internet. I've only checked the keynote while downloading and it looked perfect. However, now that I'm trying to watch any other video I'm noticing that there is no audio at all. (Not even in the mp4.part files.) Do you happen to have any idea why that is?

Environment:

xcode-select -v
xcode-select version 2354.

swiftc --version
Apple Swift version 5.0.1 (swiftlang-1001.0.82.4 clang-1001.0.46.5)
Target: x86_64-apple-darwin18.7.0

I've had the same crash(if I remember correctly) as #87 so I first compiled the script with swiftc.

swiftc -o wwdcDownloader wwdcDownloader.swift

After that this was the command I used while the pwd was my external HDD.

/path/to/compiled/script/wwdcDownloader --hd1080 --pdf --sample

Any help is greatly appreciated!

Can we use it to download subtitles

Can we use it to download subtitles

Add ability to pick videos to download

Is there a way to add ability to pick videos to download? "Download all" takes time...

Download stopped automatically after downloading few videos.

Download stopped automatically after downloading few videos.

Doesn't download 2012, 2013 & 2014 videos (hd720, or SD)

Downloaded script today to download the WWDC videos for 2012-2014, however it reports that the video is not available for either hd720 or SD.

Example output from the script:
`
[Session 600] : Debugging UIWebViews and Websites on iOS
Video : Video is not yet available !!!

[Session 601] : Optimizing Web Content in UIWebViews and Websites on iOS
Video : Video is not yet available !!!

[Session 602] : Delivering Web Content on High Resolution Displays
Video : Video is not yet available !!!

[Session 604] : Advanced Effects with HTML5 Media Technologies
Video : Video is not yet available !!!

[Session 701] : iOS Accessories
Video : Video is not yet available !!!

[Session 711] : Power Management
Video : Video is not yet available !!!

`

The videos are definitely there and downloadable. Like 2012 session 402 Working Efficiently with Xcode https://download.developer.apple.com/videos/wwdc_2012__sd/session_402__working_efficiently_with_xcode.mov

pdf format not recognised by preview

Cannot open the pdf.
Message error:
"It may be damaged or use a file format that Preview doesn’t recognize."

Needs Grand Renaming rules applied to work with Xcode 8 & Swift 3

Now that Xcode 8 & Swift 3 have been released, I want to download all the WWDC 2016 Content but the wwdc2016.swift script doesn't work with Swift 3 and the Grand Renaming. I know all the serious developers downloaded this content a long time ago but I'm a NOOB with Swift so it would really help to have access to all the WWDC 2016 content. Thanks - Russell

Examples of errors output when I run the script:

./wwdc2016.swift:53:42: error: 'NSURLSessionDownloadDelegate' has been renamed to 'URLSessionDownloadDelegate'
class DownloadSessionManager : NSObject, NSURLSessionDownloadDelegate {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
URLSessionDownloadDelegate
./wwdc2016.swift:60:21: error: 'dispatch_semaphore_create' has been replaced by 'DispatchSemaphore.init(value:)'
let semaphore = dispatch_semaphore_create(0)
^~~~~~~~~~~~~~~~~~~~~~~~~

Provide check for xcrun

I tried running this script on an El Capitan computer that didn't have XCode and received the following error:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

A quick search told me that I had to install XCode-Select, but it would've been better if the script did this check and either did the install for me or walked me through the necessary steps.

Update script call

When I tried to run "wwdcVideoPDFGet-curlVersion.sh", I've got a massage "commend not found"

"./" fixed the issue: "./wwdcVideoPDFGet-curlVersion.sh"

Downloading of sample code fails

Version: 825db59

Summary: If you try to download sample code, no sample code downloads.

Repro steps:
Try to download a session with known sample code.
./wwdcDownloader.sh --sessions 225 --sample-only // As of creation of this ticket, this defaults to 2019
./wwdcDownloader.sh --wwdc-year 2018 --sample-only

Expected result: sample code is downloaded
Actual result: no sample code is downloaded

Comments: I think that Apple has changed the format for linking to sample code. If you look at session 225, there are two sample code links, one which matches the current download link matching regex, and one that doesn't.

[curl -k] required on old systems

Crash

xcode10.2.1

Broken on ElCapitain?

Seems like the downloader is broken on El Capitain. Seems like maybe /tmp can't be written to?

Mac-mini:NewWWDC2015 sanguish$ ./wwdcVideoPDFGet-curlVersion.sh -f HD

Using 'no password' mode (this is possible since WWDC 2014 sessions addition)!
try using -l option if download does not work.

sed: /tmp/wwdc-session-10E7EB69-D9A8-4B8C-ADA8-969005B8A4C8.tmp/titles.txt: No such file or directory
mv: rename /tmp/wwdc-session-10E7EB69-D9A8-4B8C-ADA8-969005B8A4C8.tmp/titles.txt to /tmp/wwdc-session-10E7EB69-D9A8-4B8C-ADA8-969005B8A4C8.tmp/titles-to-be-escaped.txt: No such file or directory
sed: /tmp/wwdc-session-10E7EB69-D9A8-4B8C-ADA8-969005B8A4C8.tmp/titles-to-be-escaped.txt: No such file or directory
########## DOWNLOADING HD VIDEOS and PDFs files ##########
Mac-mini:NewWWDC2015 sanguish$ 

Broken after Apple Developer website came back on line?

The script can't find the video.html temp file any more. Thanks for checking this issue because I really would like to download all videos at once.

Crashes with `beyond endIndex`

Tried to run

wwdc2018.swift --hd --pdf

Crashes with the following message

Downloading HD videos in current directory
Let me ask Apple about currently available sessions. This can take some times (15 to 20 sec.) ...
Fatal error: cannot increment beyond endIndex

Tested on both macOS 10.12.6, Xcode 9.2 and macOS 10.13.5, Xcode 9.4.

Could not find a way to debug the script with Xcode yet. Any pointers?

No such file or directory error

I stopped the script when downloading the SDVideo. Again when i restarted the script, i got this message.

******* DOWNLOADING SD VIDEOS ********
I see this is the first time you go for the videos themselves! Cool :)
All downloads will go to your Desktop/WWDC-2013 folder!

WWDC-2013/SD-VIDEOs/*.download: No such file or directory

More filtering

It would be great if there would be an option to download only samples / pdf /videos

-c <content type> pdf, samplecodes, videos, all. (default all)
For example

• Download only pdf for wwdc 2015:

      wwdcVideoPDFGet-curlVersion.sh -c PDF
  

• Download only sample codes of WWDC 2014

  wwdcVideoPDFGet-curlVersion.sh -y 2014 -c SAMPLE
  

WWDC 2014 filenames are incorrect

Filenames and session IDs do not match up. Problem for both PDFs and videos.

E.g:

downloading PDF for session 206: http://devstreaming.apple.com/videos/wwdc/2014/206xxdiurnffagr/206/206_introducing_the_modern_webkit_api.pdf?dl=1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8718k 100 8718k 0 0 569k 0 0:00:15 0:00:15 --:--:-- 471k
mv: rename /Users/marius/Documents/WWDC-2014/PDFs/206 - Creating 3D Interactive Content with WebGL.pdf.download to /Users/marius/Documents/WWDC-2014/PDFs/206 - Creating 3D Interactive Content with WebGL.pdf: No such file or directory

2016 Tech Talks

Any chance of getting an update to support the 2016 Tech Talks which were just released?

how to pause download?

How can I pause downloads?
What if I starts again, will it download all videos again?

Can't download

Hi,
I'm getting the following error:

Maciej-Swics-MacBook-Air:wwdc-downloader-master username$ sudo ./wwdcVideoPDFGet-curlVersion.sh -o /Volumes/media/TV/WWDC/ -v -f HD [email protected]
Verbose mode on
Password:
Sessions to be downloaded:
Output directory: /Volumes/media/TV/WWDC/
Getting appIDKey...
appIDKey: **********************************
./wwdcVideoPDFGet-curlVersion.sh: line 27: /tmp/wwdc2013.tmp/titles.txt: No such file or directory
******* DOWNLOADING PDF FILES ********
******* DOWNLOADING HD VIDEOS ********

Any ideas? Tried running with sudo with the same result.

Request to Download Specific Videos

It would helpful if we could specify the specific videos to download instead of all the videos.

Broken at least with 2015 videos

It seems the videos have been moved (there is a 301) and also the video.html cleanup doesn't seem to work any more. I haven't tested it with earlier years.

Name suggestion

Hi.

Not exactly an issue but a better, catchier name could be DubDubGet.

Anyways, it only a suggestion.

Thanks.

Multiple copies of sample codes are downloaded if they are referred from multiple sessions.

Sample codes are being prefixed with their corresponding session IDs. While this is great to easily locate the same. it causes the script to download the same sample code multiple times.

Ex: The sample code titled 'DemoBots: Building a Cross Platform Game with SpriteKit and GameplayKit' is being downloaded 5 times.

As the above sample code is about 120 MB, it results in consuming additional bandwidth and time.

Possible solutions:
Easy : Don't rename the file.
Better : Detect the previous download and create an alias for the same.

How to use this script?

How exactly this script is supposed to be used? I did try to use it in terminal by the command and got command not found error.

$ wwdc2016.swift --hd -bash: wwdc2016.swift: command not found

Thanks

PDF for wwdc 2015 are not downloaded

Instead of the PDF file, a 16byte file is created with the right name like 217 - Adopting New Trackpad Features.pdf and the content is File not found."