Google Developers Day US - Intro to Data APIs

(My Original Blog Post: -*http://api-madness.com/post/google-developers-day-us-intro-to-data-apis-2/)
[youtube]http://www.youtube.com/watch?v=HCHZmFPXAJc[/youtube]

Attention! API Garbage (might be unreadable):

data Google actually

I remember one and my son just got her down with the dual developer programs and I ll be talking a bit today about the Google data APIs of this is going to be an introductory sections on trying cover a lot of ground in the shallow depth of production want to talk about and I get pretty excited when I start talking about is something I work with everyday so administrative reasons save some for the advanced session which is going on later today so yet with that so it started him and I think you re a little late so miniature and keep it short and that it is out for lunch so of the mentioned that I get pretty excited about the Google data APIs may not know of a lot about the APIs and I have a shorthand summary people had used the APIs or okay for him right end of so there s been a lot of buzz going on about these new services that Google is providing anyway that were making them programmatically accessible so that a quote here from one of our friends over at Yahoo and go got another one that comes was from the blogosphere gets a little more compact so people are or I have excited about this and what is it exactly that it s going on that s getting people so excited old Google has started to release a whole bunch of APIs programmatic interfaces that allow you to do interesting things using Google services we ve got to toolkits and libraries and Web services and software to be centering in on the Google data APIs and we have there that the little title and full fidelity APIs or saying when we release a full fidelity as we want you to be able to do everything you can do through the front and application but through a programmatic interface so you can write software to automate any of the tasks that you normally have to do by hand chain of sitting in front of the browser so these are the goals for Google data APIs and I wanted to contrast this with the some APIs that you may be more familiar with some of the more visual and functionally oriented presentation APIs for Ajax APIs to do things like maps or Ajax search things that you can embed in your webpage and you don t really have to do any preprocessing of the data you re you re getting something that is very visual in his couple of event to contrast that the data APIs or really about the data that sitting in our servers on the backend and so when you pulled on the stage only to format it manipulated transform it and, with some kind of presentation so this is this is where kind of the mashup comes and it s a great way to get data to combine different sources or to use these different services to simplify the process of creating a rich application so we set out to design these APIs we were trying to do several things I mention the full fidelity aspect are trying to give you programmatic access to all things you can use a browser but we wanted to do it in a way that was simple that was easy to use it was based on open standards of a new programmers hate to to read aspects documents and designs so we wanted to try and make it simple and straightforward as possible so it uses a rest-based approach to to do create update retrieved delete functionality data manipulation we needed a way to query to get the information you re looking for because you know some of these data sources can be very large if you re for example looking at all of the events in your calendar well how do you find the one you want and then once we designs this this API we wanted to be able to use it in as many different services as possible so once you ve learned how to use the Google data APIs should be able to use the same knowledge with Google base calendar spreadsheets Google apps watercooled notebook costs while the school code search and done we also wanted to enable you to do that in any language so I m we ve released a client libraries open source libraries in several languages and there actually some that launchers in the community have donated or created their open source so that will look at some you know here s our goals now how did we design this system how does it actually work and we wanted something that was simple simple data format in a simple protocol and we want to do something that was existing in open standard not try to create something new that would be confusing and in something to learn so we looked at the atom syndication format which is used for a lot of different feeds already things like blogs that it s very similar to the RSS if you re familiar with that and we also wanted a simple protocol so we went with the HUD fears underlying system using that for each UDP verbs before that post put and delete to to publish the crud operations that you need and there is an existing protocol that s tied in with the with Adam all the atom publishing protocol so via the services that were designed for G. data are designed to be fully atom publishing protocol compliant so if you know what the atom publishing protocol you can work with G. data and a lot of the tools that on already exist for the atom publishing protocol with the with little to no modification you can use them with all the services so that was kind of our starting point was the atom publishing protocol from there we had to add a few things that we needed a suit to really make this work for all of these various services so we added some some extensions on top of this atom publishing protocol s and extensible open protocol so we were free to add new things to it so we we created new data models that extends the standard atom feed and entry and we added in the query concurrency controls what happens if two people are trying to change the same data at once and we added in authentication we re working with simple HET P. but a lot of the solutions that are there for each UDP authentication are fully secure so we ve designed some new things we had to roll around for him so going to go through these different extensions that kind of high-level not really digging into a lot of details and later today at the advanced session will be going through a few more things so before we go any further a can of been talking about this data format and thought it would be important to show it so we ve got a whole lot of XML here this is one entry so atom publishing protocol deals with two large XML constructs feed and entry sore feet is a collection of entries entries have all kinds of information on all represented as XML and this happens to be a recipe actually Google bases is for you can store all kinds of different information and metadata about things so one of the popular item types in Google bases recipe so here s an example we have any recipe and I ll try to highlight kind of the human readable parts right deserves a lot of the XML parsing needs to be done were actually going to be looking at a lot of the XML throughout this thing so of you know it s the details aren t all that importance of interest have to kind of get a feel for how the data is laid out notices the data API there s not a lot of flashy front and type stuff it s mostly in a dealing with large amounts of XML so let s begin now with authentication so we created two different ways to authenticate with the Google data services there was a client based where maybe you have a desktop application something you ve installed where you trust a system to give it your username and password so you login and then there was a browser-based system we re using redirects to send you to sign into page and then send you on to a application Web application both of these are token-based schemes so the sign is secure and menu gets a token which is used and expires in November vote in all those kinds of things so that only are token is exposed through the HUD contract alright so then moving on Aquarius at Google we like queries and so we ve come up with lots of different ways to query your data we have fulltext search categories you can query on things like the updated time when an entry is changed then we have the custom ones a lot of the different services that have implemented Google data APIs have created specific services like that example bears from spreadsheets you can query for all of your all the rows in your spreadsheet or you can just ask for between this roadless row on Google base actually has kind of gone all out and created their own prayer language which is a pretty complicated and do some really interesting things with also you can specify different output formats protocol itself works using atom but it when you re pulling your information you can get it in a variety of forms whether it s RSS or juice on you can also do a chase on and scripts like a Tucson P. type callback into a JavaScript application so there s lots of options and you ll have to dig through the documentation to look at you now the difference is in how all these things work but we ve tried to really make a rich query infrastructure were you can get just the data you want so now its talk about the data model is shown in example before this is a different entry this one comes from calendar and what I wanted to show us how we extended the data model so for example this is some this is the XML that comes from the atom standard so we have an entry it has an idea hasn t updated time there s a title and content and in addition we added several things so for calendar specifically there s quite a few are things we have reminders we have a time we have status of the event so I was really nice in this regard that you can just plug in XML you can just add whatever else you want and actually some of the services allow yourself to define new elements and the service will just take those and store them in index and allow you to search through those as if they were part of the standard itself I mentioned the concurrency so we use a system of optimistic concurrency since this is a rest-based arrests title Web service we give each item a unique URL in addition to that we have an edit your all or an edit link which has a version number appended to the end of it so if you and your friend are both pulling down the unit of different versions of say your spreadsheet and he updates his cell and your data so that it works out so that you know you re both doing it at nearly the same time it because it s in their first view of overwritten when did the version number will changed after your friend did did do is update so you ll actually get paid up for a nine conflict air I m saying you know this version number change in the pulldown menu version to be able to change it closes with a pretty simple way of solving this problem when it was easy-to-understand easy-to-use another thing we added were treated at times so we have were using the entry for example over and over in all of these different services they each have their own flavor of entry so you know if this you are aggregating feeds and you were looking at funerals these different entries how do you tell which of your entries came from calendar spreadsheets or blogger so we added in kinds of ways of categorization using links actually schema links so we also use this for some of the XML attributes or properties elements that are reused over different services over examples of numbers there at all notes can small for people in the back but down there to phone numbers and one is labeled as a work and the second is labeled as a pager using that scheme your so now here s an example if you wanted to see for example calendar and look at all the entries you would just go to that URL you get an atom feed and the there are of namespace extensions used identify semantic entities and it items not easily knock one out of the RSS so it assesses as an atom feed some of the browsers for example are built to understand how to interpret the data like as if it was a blog of weapons for example so we ll see here is her feet and holiday we ve got all these different entries for different holidays okay so here s some of the services that the tie into Google data APIs on highlighted Google spreadsheets down there at the bottom because I m going to become walking through an example using spreadsheets have show you how this actually plays out so our goal here for this simple example is to change the contents of her road in a spreadsheet so the first thing we have to do before going to be changing a row going to have to authenticate because of their permission controls on your spreadsheets you can say who can and can t edit the spreadsheet so you just login and the system will actually be able to show you can actually query for a list of all the spreadsheets that you re authorized to edit for example so this is the spreadsheet were going to be changing only small and simple were looking at the hours that people have worked in the items they created and in those hours and then we ve calculated items per minute out there so something produce so authenticate were going to need to send her information I ve chosen to do a client login example so client login is done over a HUD PS using a post so there s an example of the information you send my highlighted the service name down there when you authenticate your syndicate for one service at a time so they each have kind of code words that we ve chosen arbitrarily people ask me why the spreadsheet services named wise and I have no idea okay so once you re dedicated yet your token back and then you use that token you included in the HUD header of all of your following requests so the system will check to make sure that you re allowed to see the spreadsheet there is an example were looking at changing a row so we re going to be pulling down list feed you can see down there this is our get URL of feeds list the SID is a spreadsheet ID which is this really long ridiculous string and worksheet ID identifies a particular page within worksheet number going to be looking at the private and full representation of the data others have a lot of information that URL there and I m not going to get into it right now but I m basic ideas of their lots of ways that you can ask for your data to be represented you can view a spreadsheet as a series of cells or as a series of rows were choosing to look at it as a series of rows and there are different levels of detail you can ask for centuries so here s our results this is where spreadsheet looks like it s a big mess of XML this is why we have XML parsers so if we cut out some of the stuff that were not interested in maybe some of the namespace declarations and things like that we get something a little more readable but here s the data that we really care about right when were were dealing with changing a spreadsheet so if your member back to the column names we had our spreadsheet we do the first row in the spreadsheet had name hours items and the IPM items per minute so those are then translated in the list feed into custom XML elements so there actually names according to the first row spreadsheet and then you have the contents there of those two different roots to data roots so we have via the edit URL and mentioned before that you know we were using optimistic concurrency so we need to pay attention to the edit link for going to be changing this data so you can see that the at the ends says you know Pvt.: then there s a few random strings of identifying the person so now that we have changed our information we actually send back an entry we do output to that edit link and that s the data were sent to the funny thing is we didn t actually change anything that we did update the row just doesn t look any different so that was a short example of a wanted to kind of show a few of the projects that are using our APIs because of it I wanted to give this kind of a feel of the basics on how how you do a enough some of the common operations but so when she realized you know that you have these resources the services and you have a simple way to do no custom often up operations scripts things or a no manipulate data there s a lot of possibilities so people come up with some interesting things on our oh website PIC data we actually have several him highlighted projects that people of created and then if you look through all the different services most of those also have example projects things that developers in the community have been creating a so for example here s a list of applications that are using Google base slot of these cover a wide variety of different functionality and are interested in different types of information on why also wanted to show a couple of screenshots some that are out at an interesting talk very briefly about and the clock is actually a system where you can plug in your calendar and it ll create a graphical representation of what you re busy free time and can avoid think they re either planning to make a clock or something to me it looks like right now you can get it as a gadget and others in different ways to to use it but that would be really cool if some of these things translated into physical devices separate from your computer found there s also like an office kind of planning application called timesheets which attached the other spreadsheets and how it renders some of photo editors that use the top Picasso web service so who see it looks I ve are doing okay I m time I wanted to do a simple dental I mentioned some of the client libraries that that we have and wanted to show you how easy it can be using some of these tools so hopefully you can see this actually might be kind of hard on the back but I m going to be using a python of the Google data library I have a personal interest in showing this group s see this is what you call like ultra extreme or ramming right because I don t have one person looking over my shoulder I have like 200 so that always makes you feel it their review seven the import fails okay so I have imported the library found for spreadsheets and also I wrote a little helper module that does things like it s in my username and password so you guys don t have to seal that ends some of us start by creating a clients in the number going to have my hope rewarded okay so I mentioned before of doing a doing a query just asking for all of the rows in my spreadsheet so I m asking for a list feed and I m going to give it the spreadsheet ID which I ve also hardcoded into helper module okay so looks like feed works and if I print out my feed of the a lot of XML so that s good him so that I can look and see you know how many entries are in my fee okayed virtue which if I look over at my spreadsheet here yes there are in fact two euros and then if I go wanted to for example change one cell I could look at the soles feet again giving it the spreadsheet ID okay print this feed it s even longer and if we look at the length here to see how many entries are in the see that there are 12 so if you look at this regard the of 4 x 3 so it gives you the full cells that have been so down when you change one of these and the I mentioned that I m a changed ours because the items per minute this is calculated so as I change that you should see the items from that change so were going to change cell to grow to and returns the hours to 15 okay looks like it worked and we should be seeing this change appear if it doesn t trouble hopefully I m looking at the same spreadsheet that I m editing that might be the problem actually because it looks like it did change you lapse a bummer at of a good try and find it but I m worried that it would just drag on when I open up for questions and at the end all the while proof that this actually works a no actually it s a bit. Dates now the question was I don t need to refresh the spreadsheet and the I might try and see usually need to out I did I did need to okay a thank you excellent excellent question alright so we have some microphone set up for questions if you fool would mind stepping up to those and I m which she embodies the checksum will for is their checks for Unicode and the pursuit of its most Unicode is a physical Unicode assumes that I could take from an English US hosted Google data service" Hindu you were Japanese or Ezra or any Unicode character to those values and have come back intact You actually tried that out a few weeks ago and so I know it works for rule-based beta works for everything you so the question assumes a well rooted deep deprecated video Google search souls API is going to use tools for the ACCPAC Ajax API so you out if you deprive to the reverse of ability to manipulate that this thing off the search results but now he is out this of Google data APIs so used to about a developers to benefit this of some results are so could this explain the difference why you choose to restrict it to spend one set of tools that open up for the vessels of I m actually not sure what why things at the minute we decided to do things a different way in this respect I think part of the reason that we allow people to manipulate the data here is that it s their own data you know this is information in my spreadsheets and I need to be able to get the data I m interested if I of cushion at the very beginning you mentioned the associated APIs as full fidelity it is so clearly to involve their right to get everything you need to do to you is that his is the stated goal for all users to visit for them to be feature complete as for your yes so that is the goal of a think a lot is left up to the product teams or you know developing the whole system as to what makes sense whether all this is definitely to allow you to do as much as possible through the API free from having to sit in front of a browser that didn t do this for some of the okay to break the lesson so I waited using something like this and our ego but I haven t gone on some high-traffic website of the one-way is always hard to make sure that there s enough performance on the other and add it to get rid of response and not finding my website or something on you all is it responsible to the office if I got to go somewhere that it is the response of your stuff if I had to do with timeouts and think about four or does the is the API good enough to regard as a response to the gas so I don t think there really guarantees on things like in a response times timeouts service level type agreements I think you should probably worry about timeouts although of course we were always trying to make sure that doesn t happen to you right where a working to speed these things up as much as possible make sure the scale but the API itself is just a GDP price of its were trying to keep it simple and so there s not really anything built-in that would protect you from things like HTP timeouts you was her okay okay Rigo a lot of work on media or with feet Google ad words if you are basic and if there is a solution to a problem of up to version in beta and granularity of and stability... which is to change it pretty much constantly of which is hard for developers to keep up with the goods of life of what as a theme before Google and/or the community in general do see for kind of the architecture of domain model you expose how quickly version to know our stuff I work in a slowdown why don t think we probably one of slowdown is as difficult as that may make life I m not sure on our broader grand strategy of the kind of how we re trying to address this problem we are aware that it is an issue that you know having people run to catch up is and isn t a good thing but you know at the same time we don t want to hold back functionality so it s a difficult problem and I m not really sure kind of our strategy or direction of the whole s success will follow other yellow so in general is in terms of organization within the company is their elected in API specialist group and they could ve put up its practices are adopted or to think of it as a crosscutting across all be different if Edwards was faced with maps and look for stuff workers to select what s your role in writing the sure a so it s a little of both kind of as far as you know dealing with some kind of crosscutting support organization for creating APIs and you know and then letting each team do their own thing so my role is actually two or put a lot of the developers who are using these APIs and help them understand how it works developed tools create examples in a dinosaur I don t know with the needs of few wanted to jump on that one we ve got a member who s going to be giving out the advanced talk later today so so a seat at our office to care for the faster data talks with Richard Seifert felt confident on the minute she is getting her stuff for you to Google I had a question about how this is Google and where did you get a specialist so we were developing the do you protocol we see it that way and to see so different approaches taken with producing the Edwards vigorously different API and there s a lot of cross collaboration with tapes out we also are very much aware of the issue of a feature we don t break existing clients we would make sure that it preserves much of XML to look that right now I m moving forwards were very conscious of the states where it was a bonus was greatly cheaper which is in a follow-up story that notes are the resident judge on the second microphone that will submission about Yahoo pipes O. ability to new data compared to G. data with no rights and sector doing stuff that uses bold about that in no art are they using the protocols were now so I m not really a pipes expert so let me preface it that way but I know that prices involved in accepting fees and allowing you to manipulate the data and present a new feed up on the output so they actually have used a lot of the Google data feeds it speaks atom format and it can also resent an RSS feed which you can consume and pipes so pipes actually has a plug-in or a recall of the data source for Google base so I know that they been no using a lot of G. data things do that so I think that some do they work well together as one does JAG 18 first brainstorm about the ethical authentication mechanism YTD discard the 90 right of God not really sure actually I ve looked a little bit open ID and I don t really know death of a not sure that it solves all the problems we need harder for well no one ago into it because I really don t know well enough to be able to speak intelligently about Sirius either of those places CD and it clocked up there because I m the guy who wrote it on this exile is this one I actually have a lawful model here is a little eight iron and is followed like a disaster tries to take it off the unit but interesting places like Richard is so infinitely good and since they don t know what to do with it or to put it on the news that s come Grammy thing too is the clock is actually written in Ruby and I noticed I was enlisted as a private supported languages are not sure just how actually these advance slides I ve seen this up for a list of all of the languages that either people are contributed or know people at Google s in the bar suit the skewed so we mentioned earlier on the search so that you have reciprocated some point which I was usually a popular with people who do using him what assurance do I have as a developer using the G. data if you specifically for calendar because you do the same thing won t happen so that it admitted a precocious thought it would community you have looked at these APIs and then asked you was the economic value to Google there are ads going through them right it seems like the right thing to do but you know that s not always a good reason for keeping supplies for our death or so of me talk a little bit about the the motivation and why and I wanted to provide the services you we ve created the services that we think are useful a lot of fun and I know we ve created web applications around them like for example calendar right but there are lots of interesting things that you can do with calendar that we haven t thought of or that we haven t had the time to to put out a release so by creating these APIs then we can greatly improve the user experience as far as providing new features right because now you can download an application that will that will do something new that our product doesn t do or you can combine them in interesting ways like there is an example that was recently released that some takes a spreadsheet of the People s birthdays like renaming a birthday and a link to a picture and then creates entries in your calendar for those people with embedded photos of stuff soak in notes kind of a neat way to allow people to create useful things that we have a lot of good answers and looks like were out of question so let me just say over sorry one more solos are so absolutely follow up on him so great APIs empower us developers to come up with cool new features that make the experience of calendar that so produces better so take the next step is very strong economic value in that activity to Google does it like a lock in users into using calendar somehow it is Google for this is because Google could see strong economic value from locking people into our cool little kids is is that we rely on the Google getting a graduate that s the one with the new dialog was just fun is resulting from the detectives were all both sound good to me a I m actually not sure about the kind of the broader nation needs you yes please oh I ll try to answer questions about your upward where the engineers will make all the decisions for it asserts a fiasco for a return to value to our APIs to make sure developers are the winners and a fundamentally speaking at the effort to talk about the Senate heard of any high level of it was accompanied by the talk law but that uses only the and open empty to work with business needs that they could figure to go for us and the absolute winners are the winners are good users if they have that data if it s all XML they could take that video to the products they consider nice with another killer piece of paper for a long lease that is the point of the question that you will write so the question was a does Google use these these APIs is Web services internally will again see for calendar exactly a bogey of we do use the bed API internally yes I know there are groups are using the APIs as far as I m calendar goes you you may have heard of Google apps right were providing all these Google webpages and services through your own domain so a that system ourselves so yeah there s a lot of internal use for the data APIs is especially kind labor is is sometimes found some limitations that I thought they are a little on second values to the political will in the future API to manipulate link to add those are better like getting second one and send with nothing but galvanizes into closets but I cannot did he doesn t think okay you re talking about spreadsheet is typically a so for a okay so you re saying column manipulation using he out as those federal agents went to Louisville on a set values to the little one needs at a actually think you can clear a row by sending him aerobic update but not I think right and shift the other is a sure path so I guess that s one of the limitations will have to go to work that hard things okay so of living to say that the that so will we have some Google data station some of a lot of tops out by the beanbag chairs and there s going to be a group of us in in the black T-shirts answering questions or if you want to come talk to us feel free to drop by and thank you very much