These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.07 Sep 2017
I am currently crafting a set of services as part of my Human Sevices Data API (API) work. The core set of services for organizations, locations, and services are grouped together as a single service, as this is what I was handed, but all the additional APIs I introduce will be bundled as separate set of individual services. Over the last couple of weeks I’ve introduced seven new services, with a handful more coming in the near future. I’m enjoying this way of focusing on services, over the legacy way that is very resource focused, as I feel like it lets me step back and look at the big picture.
When I was defining the core API for this work I was very centered on the resources I was making available (organization, locations, and services), but once I took on a service mindset I began to see a number of things I was missing. With each service I find myself thinking about the full life cycle, not just the APIs that deliver the service. I’m thinking about the easy ones like design, deployment, and management, but I’m also thinking about monitoring, testing, and security. Then I’m delivering documentation, support, communications, and thinking about my monetization strategy, and access plans. I’m not just doing this once, I am thinking about it in the context of each individual service, as well as across all of them, taking care of the business of the services I’m delivering, not just the technical.
While some folks I talk to look at some of this as repeat work across my projects, I just see them as common patterns, that I should be reusing, refining, and delivering in consistent ways. I’m thinking about delivering the technology in a consistent way, and the operational, but I’m beginning to think about education, training, and how I can help folks on the provider and consumer side of things learn how things are working. I’m not just doing the technical heavy lifting to deliver APIs and then walking away, I’m bundling each search with what is needed to be valuable and successful as an actual service, that is API driven from start to finish. The service is accessible via an API, but it is also delivered, managed, and supported using APIs–everything has an API.
The Human Services Data APIs (HSDA) I am delivering aren’t just a single API, or set of service. They are an open source set of services that I’m putting out there for others to adopt and deliver as part of their own operations. I don’t want these to just be plug and play APIs, and want them to be plug and play services that deliver the information people need to find vital services in their community. Thinking of my APIs as services, and breaking them up into independent microservices helps me address the technical, business, and politics of delivering the technical components cities and organizations are needing. I’ve been pushing the business and politics of APIs since I’ve started, and trying to doing things in as small pieces as I can since the beginning, but the microservices conversations I’ve been tuning into have helped me think beyond the tech, the size, and actually consider how I’m doing this to deliver services to humans–it just an interesting twist that my primary project is all about delivering human service microservices. ;-)
You are full steam ahead with your microservices campaign. You’ve read Martin Fowlers blog post, and talked about the topic with your team for the last six months. After a couple pilot projects, you are diving in, and have started decoupling the monolith of systems that you depend on to operate your business each day. You have mapped out all the technical details of all code, and backend systems in play, and have targeted about 30% of existing systems for reworking using a microservices strategy. Yet, despite all your planning, your microservices effort will still fail just like your API efforts, and its predecessor the SOA initiative did.
Despite all your research, planning, and eye for the technical detail in about 7 months everything will begin to slow, and by month 10 you will begin to get very, very frustrated. You see, you haven’t included any of the human element in your planning. You have thought about the business, cultural, legal, financial, and other non-technical aspects of operating your systems. You’ve done a fine job of developing a strategy for decoupling your monolith database, but you haven’t invested any time in what it will take to educate and bring up to speed all the business users, support, QA, and other folks you will need to have up to speed to actually make all this work. You are so blinded by technological trends, you forget to spend time talking to the people.
You are feeling good right now because you have surrounded yourself with yes men. People who are in perfect alignment with what you are doing. They think like you, and have drank the same kool-aid. However, once you start implementing your grand strategy to make every smaller, more micro, you will begin to see friction. Sales folks will begin to see you as a threat as you fragment their landscape, and be seen as taking a piece of their action. Business users will begin to freak out, because while they had to battle with IT before to get access to resources, now they have deal with man many smaller teams, with even more unfriendly people to talk with–you just increased the scope of what they need to know. Many will see your work as just increasing complexity, and expanding the landscape of their work, in an already hectic world. As this perception increases they will find ways to throw a monkey wrench into your finely tuned plan.
Your SOA rollout failed because it was to heavy handed, dictated everything, giving vendors too strong of a voice, while also strengthening IT too much in the eyes of business groups. Your API efforts underestimated the fatigue management and businesses users experience from SOA, and your team never really believed in API anyways–they just thought it was a fad, trend, and pipe dream. Now your team is sold on it being all about offering up smaller services, and you are paying lip service to fact that “service” should be including more than just the technical, but you actually haven’t done the heavy lifting. You don’t have your training, support, documentation, QA, testing, discovery, terms of service, security and other critical aspects of each service in place, let alone the comprehensive view of how this works across the thousands of microservices you are planning. Let’s get real, your microservices plan will fail just like API, and SOA. You are just too in love with the technical, and willfully blind to the human side of all of this to ever make it work.
Note: If my writing is a little dark this week, here is a little explainer–don’t worry, things will back to normal at API Evangelist soon.
The Human Services Data API I have been working on is about defining a set of API paths for working with organizations, locations, and services that are delivering human services in cities around the world. As I’m working to evolve the OpenAPI for the Human Services Data API (HSDA), I’m constantly mindful of bloat, unnecessary expansion of features, and always working to separate things by concern. My thoughts have evolved this due to a hackathon I attended this week in San Francisco where a team at Optmizely worked to decouple an existing human services application from its backend and help teach it to speak Human Services Data Specification (HSDS)–allowing it to speak a common language around the services that us humans depend on daily.
As the hackathon team was decoupling the single page application (React) from the API backend (Firebase) I took the API calls behind and published to Github as two JSON files. One of the files was locations, which contained 217 human service locations in San Francisco, and metadata, which contained a handful of categories being used to organize and display the locations. In this situation, there is no notion of an organization, just 217 locations, offering human services across five categories. This legacy application, and forward engineering hackathon project was quickly becoming a microservices project, ironically it is a microservice project that was about delivering human services. ;-)
Looking at this unfolding project through a microservices lens, I needed to provide a single service. In the context of Link-SF, the original project, I needed to offer a service that would deliver 217 locations where people can find human services in the areas of food, housing, hygiene, medical, and technology via an web, or mobile application. To help me achieve my goals I began to step through each of the steps of the lifecycle of any self-contained microservices:
- Github - Each of my services begins with a Github repository, so I created a new repo.
- Definitions - I defined the service I wanted as an OpenAPI.
- Design - I worked to keep the design of it as simple as I possibly can.
- DNS - I relied on Github’s DNS for this service, but may setup my own subdomain.
- Database - I published a YAML data file into the data folder for the Github repository, acting as the database for my service, leverage Github and Jekyll for helping me broker database connectivity.
- Deployment - I deployed a simple static JSON API driven from the locations YAML store.
- Management - I am using Github as the management platform for my API, helping me deploy and manage consumption of my service. I’m using the Github API as the programmatic layer for managing my service operations to help me continuously deploy and integrate with this service.
- Portal - I am using Github pages as the portal for my service, providing both human and programmatic access to my service, making human service locations more available.
- Documentation - I published static HTML, and Swagger UI documentation for the service.
- Support - I have published a support page, and will be providing support via Github issues, Twitter, and email.
- Communications - I have a blog published, and will be publishing the story of the service using the Jekyll CMS blogging solution.
- Caching - The API, portal, and all aspects of the service I’ve launched is being cached by Github, riding on the backs of their Content Delivery Network (CDN).
- Reliability - While Github has been known to have stability issues, it is still some of the most reliable way to host data and API driven services.
- Encryption - I’m leveraging the Github DNS, and using their encryption in transit by default. I will be using my own subdomain and encryption in the future.
- Security - I’m offloading platform security to Github. They have more resources than I do. I’m also looking to keep things more secure be keeping my services as static as possible.
- Testing - I am setting up a series of monitors to ensure the service stays available, and delivering expected data and promised schema. I will be publishing a status update and history page.
- Transparency - Everything involved with my service runs on Github, either as a public or private repository, with the entire lifecycle transparent publicly, or privately to whoever is given access to the repository using Github.
- Observability - The entire service runs on Github. The entire lifecycle is observable in the Github history, and leverage their existing infrastructure for identity and access management (IAM), and continuous integration and deployment.
- Discovery - There is an APIs.json available in the root of the project, providing a machine readable index of the schema, data, API, and other resources available via this service.
I’m calling my new service, born out of the Link-SF legacy application Human Services Link. However, the only thing remaining of the Link-SF application is the data. I’ve forward engineered the schema, API, and web interface to all run on Github as a single, yet decoupled service. It is a reduction of the overall Human Services Data Specification (HSDS) schema, focusing in on just providing services at a handful of locations, reducing complexity and scope, to deliver a specific service–do one thing and do it well. I will be launching an Amazon EC2, and containerized version of this human services microservice, but I wanted to look at this one incarnation of it though the microservices lens. Sure, its not your classic microservices definition, but I think it holds up to a microservices test, it just makes some different architectural choices than the rest of the community might have made.
My human services microservice definition solves a single problem, for a specific type of end-user. It was originally geographically limited to San Francisco, but with with this evolution I’ve made sure there is a state/province field, as well as country, so that solution can be deployed for any city. I want the compute, database, DNS, API, portal, docs, and UI elements and admin tools to all be self contained for the delivery of a single microservice, but I wanted it all forkable, so you could launch this service over and over, in any city around the globe. There is still a lot of work to be done on this project before it is ready for prime time, but I’ve planted the seeds when it comes to delivering microservices in my world this way. I’m hoping it will be something I can’t ignore, and will keep pushing forward as part of my overall vision of how we can deliver sustainable API driven services.
I really like the static nature of this approach. I really like the forkability of this approach, and how you can use Github to separate out concerns by organization. I really like how I can offload much of the operation of my backend to Github, including security, CDN, and scalability / reliability. It is definitely not a pattern everyone should be following when doing microservices, but it does provide a static forkable pattern that can really help keep services very focused and self contained, but available to work in service of a bigger architectural picture. I’ll keep playing around with this approach to delivering human service microservices, and see what I can push forward and make stick. I’m hoping it can providing a low coast way for cash strapped municipalities to better deliver up to date information about human services to their entire population.
I’ve been looking at my human services API work through a microservices lens, triggered by the deployment of a reduced functionality version of the human services implementation I was working on this week. I’m thinking a lot about the technical side of decoupling services using APIs, but I want to also take a moment and think about the business side of decomposing services, while also making sure they are deployed in a way that meets both the provider and consumer side of the business equation. My human services microservice implementation is in the public service, which is a space where the business conversation often seems to disappear behind closed doors, but in reality needs to be front and center with each investment (commit) made into any service.
Let’s take a moment to look at the monetization strategy and operational plan for my human service microservice. Yes, a public data microservice should have a monetization strategy and plan for operating and remaining sustainable. The goals for this type of microservice will be radically different than it would be for a commercial microservice, but it should have one all the same.
- Monetization - How am I evaluating the investment into this project alongside any value that is generated, which I can potentially capture or exchange some value for some money to keep going.
- Acquisition - What did it take for me to acquire the data and skills necessary to make the deployment possible.
- Development - What time was invested in setting up the platform, developing the schema, data, definitions, code, and visual elements.
- Operations - What does it take to operate the service? Maintain it, answer questions, provide support, and other realities of providing an online service today.
- Direct Value - What are the direct benefits of having this service available to people looking for human services, or organizations looking to provide human services.
- Indirect Value - What are the indirect benefits of having this service available, like increased conversation around human services, or maybe traffic and awareness of the Open Referral organization.
- Partners - What partnership opportunities are actively being sought out, or will be opened up by having this service available.
- Reporting - What type of reporting is necessary to operate and monetize this service, from tracking page views to understanding who is integrated with the data, and consuming data via APIs, or possibly through continuous integration of the Github repository.
- Plan - What is my plan for making this service available to the public, partners, or maybe internally use across my own projects.
- Elements - My human services location API is designed to be publicly available, forkable, and reusable by anyone.
- Limits - There are no limits on how each human services microservice can be used, or forked and reused. Ideally, any implementation provides attribution, and acknowledges the source of framework or data, but there really are no rules.
- Metrics - I am measuring unique page views on each page, including of the machine readable YAML, JSON, and other formats. I’m also tracking on stars, forks, and commits for each service.
- Commercial - Providing a clear track for commercial vendors to understand that the project needs their support, and can be improved upon, and evolved through their underwriting and support.
I need to have a coherent snapshot of what I’ve invested into each of my service. I need to have a base plan for how I will be executing the business side of this service–even if it just making something available for free. There are two dimensions to this conversation: 1) My view as the creator of this service 2) The view of folks who fork and implement as a service. Both dimensions should have a monetization snapshot, and a plan for executing within this business snapshot. I need all of this decoupled from any other service I am offering, but ideally they all use a common set of reusable patterns, just like the technical aspects of my microservices effort.
Just like needing the compute, database, DNS, and other technical layers to be stable and scalable across my microservices, I need the costs associated with these elements predictable and affordable. I need to know what it costs to define, design, deploy, manage, and deprecate my services. I need to know the best path forward for making them public, keeping them private, and being honest with commercial partners about the value that is being generated, both directly and indirectly. I need a way to report my costs, and revenue across hundreds or thousands of these services. I need to be able to scale this both horizontally across many services, but also vertically for single services which get deployed over and over, reused and continuously integrate wherever they are needed using Github.
I’ll keep applying this model across my human services project. I’m thinking that I’m going to be developing a whole buffet of human service microservices that run a 100% on Github like this, but I am also playing around with other varietals that run on other popular cloud platforms as well. I’m not one to subscribe to any particular API dogma, I’m just looking to explore what is possible, and do all of it as cheaply as I possibly scan, growing the number of projects I’m able to tackle. Of course, none of this would be possible without my partners funding my work, and helping me connect the technical and business aspects of doing API Evangelist.
Reducing Developers To A Transaction With APIs, Microservices, Serverless, Devops, and the Blockchain08 Aug 2017
A topic that keeps coming up in discussions with my partner in crime Audrey Watters (@audreywatters) about our podcast is around the future of labor in an API world. I have not written anything about this, which means I’m still in early stages of any research into this area, but it has come up in conversation, and reflected regularly in my monitoring of the API space, I need to begin working through my ideas in this area. A process that helps me better see what is coming down the API pipes, and fill the gaps in what I do not know.
Audrey has long joked about my API world using a simple phrase: “reducing everything to a transaction”. She says it mostly in jest, but other times I feel like she wields it as the Cassandra she channels. I actually bring up the phrase more than she does, because it is something I regularly find myself working in the service of as the API Evangelist. By taking a pro API stance I am actively working to reduce legacy business, institutional, and government processes down and breaking them down into a variety of individual tasks, or if you see things through a commercial lens, transactions.
Microservices A microservices philosophy is all about breaking down monoliths into small bite size chunks, so they can be transacted independently, scaled, evolved, and deprecated in isolation. Microservices should do one thing, and do it well (no backtalk). Microservices should do what it does as efficiently as possible, with as few dependencies as possible. Microservices are self-contained, self-sufficient, and have everything they need to get the job done under a single definition of a service (a real John Wayne of compute). And of course, everything has an API. Microservices aren’t just about decoupling the technology, they are are about decoupling the business, and the politics of doing business within SMB, SME, enterprises, institutions, and government agencies–the philosophy for reducing everything to a transaction.
Containers A microservice way of thinking about software that is born in the clouds, a bi-product of virtualization and API-ization of IT resources like storage and compute. In the last decade, as IT services moved from the basement of companies into the cloud, a new approach to delivering the compute, storage, and scalability needed to drive this new microservices way of doing business emerged that was called containers. In 2017 businesses are being containerized. The enterprise monolith is being reduced down to small transactions, putting the technology, business, and politics of each business transaction into a single container, for more efficient development, deployment, scaling, and management. Containers are the vehicle moving the microservices philosophy forward–the virtualized embodiment of reducing everything to a transaction.
Serverless Alongside a microservice way of life, driven by containerization, is another technological trend (undertow) called serverless. With the entire IT backend being virtualized in the cloud, the notion of the server is disappearing, lightening the load for developers in their quest for containerizing everything, turning the business landscape into microservices, than can be distilled down to a single, simple, executable, scalable function. Serverless is the codified conveyor belt of transactions rolling by each worker on the factory floor. Each slot on a containerized, serverless, microservices factory floor possessing a single script or function, allowing each transaction to be executed, and replicated allowing it to be applied over and over, scaled, and fixed as needed. Serverless is the big metal stamping station along a multidimensional digital factory assembly line.
DevOps Living in microservices land, with everything neatly in containers, being assembled, developed, and wrenched on by developers, you are increasingly given more (or less) control over the conveyor belt that rolls by you on the factory floor. As a transaction developer you are given the ability to change direction of your conveyor belt, speed things up, apply one or many metal stamp templates, and orchestrate as much, or as little of the transaction supply chain as you can keep up with (meritocracy 5.3.4). Some transaction developers will be closer to the title of architect, understanding larger portions of the transaction supply chain, while most will be specialized, applying one or a handful of transaction templates, with no training or awareness of the bigger picture, simply pulling the Devops knobs and levers within their reach.
Blockchain Another trend (undertow) that has been building for sometime, that I have managed to ignore as much as I can (until recently) is the blockchain. Blockchain and the emergence of API driven smart contracts has brought the technology front and center for me, making it something i can ignore, as I see signs that each API transaction will soon be put in the blockchain. The blockchain appears to becoming the decentralized (ha!) and encrypted manifestation of what many of us has been calling the API contract for years. I am seeing movements from all the major cloud providers, and lesser known API providers to ensure that all transactions are put into the blockchain, providing a record of everything that flows through API pipes, and has been decoupled, containerized, rendered as serverless, and available for devops orchestration.
Ignorance of Labor I am not an expert in labor, unions, and markets. Hell, I still haven’t even finished my Marx and Engels Reader. But, I know enough to be able to see that us developers are fucking ourselves right now. Our quest to reduce everything to a transaction, decouple all the things, and containerize and render them serverless makes us the perfect tool(s) for some pretty dark working conditions. Sure, some of us will have the bigger picture, and make a decent living being architects. The rest of us will become digital assembly line workers, stamping, maintaining a handful of services that do one thing and do it well. We will be completely unaware of dependencies, or how things are orchestrated, barely able to stay afloat, pay the bills, leaving us thankful for any transactions sent our way.
Think of this frontline in terms of Amazon Mechanical Turk + API + Microservices + Containers + Serverless + Blockhain. There is a reason young developers make for good soldiers on this front line. Lack of awareness of history. Lack of awareness of labor. Makes great digital factory floor workers, stamping transactions for reuse elsewhere in the digital assembly line process. This model will fit well with current Silicon Valley culture. There will still be enough opportunity in this environment for architects and cybersecurity theater conductors to make money, exploit, and generate wealth. Without the defense of unions, government or institutions, us developers will find ourselves reduced to transactions, stamping out other transactions on the digital assembly line floor.
I know you think your savvy. I used to think this too. Then after having the rug pulled out from under me, and the game changed around me by business partners, investors, and other actors who were playing a game I’m not familiar with, I have become more critical. You can look around the landscape right now and see numerous ways in which power has set its sights on the web, and completely distorting any notion of the web being democratic, open, inclusive, or safe environment. Why do us developers think it will be any different wit us? Oh yeah, privilege.
404: Not Found
While I track on what is going on with visualizations generated from data, I haven’t seen much when it comes to API driven visualizations, or specifically visualization about API infrastructure, that is new and interesting. This week I came across an interesting example in a post from Netsil about mapping microservices so that you can monitor them. They are a pretty basic visualization of each database, API, and DNS element for your stack, but it does provide solid example of visualizing not just the deployment of database and API resources, but also DNS, and other protocols in your stack.
Netsil microservices visualization is focused on monitoring, but I can see this type of visualization also being applied to design, deployment, management, logging, testing, and any other stop along the API lifecycle. I can see API lifecycle visualization tooling like this becoming more common place, and play more of a role in making API infrastructure more observable. Visualizations are an important of the storytelling around API operations that moves things from just IT and dev team monitoring, making it more observable by all stakeholders.
I’m glad to see service providers moving the needle with helping visualize API infrastructure. I’d like to see more embeddable solutions deployed to Github emerge as part of API life cycle monitoring. I’d like to see what full life cycle solutions are possible when it comes to my partners like deployment visualizations from Tyk and Dreamfactory APIs, and management visualizations with 3Scale APIs, and monitoring and testing visualizations using Runscope. I’ll play around with pulling data from these provides, and publishing to Github as YAML, which I can then easily make available as JSON or CSV for use in some basic visualizations.
If you think about it, thee really should be a wealth of open source dashboard visualizations that could be embedded on any public or private Github repository, for every API service provider out there. API providers should be able to easily map out their API infrastructure, using any of the API service providers they are using already using to operate their APIs. Think of some of the embeddable API status pages we see out there already, and what Netsil is offering for mapping out infrastructure, but something for ever stop along the API life cycle, helping deliver visualizations of API infrastructure no matter which stop you find yourself at.
404: Not Found
I am finally getting the time to invest more into the rest of my API industry guides, which involves deep dives into core areas of my research like API definitions, design, and now deployment. The outline for my API deployment research has begun to come into focus and looks like it will rival my API management research in size.
With this release, I am looking to help onboard some of my less technical readers with API deployment. Not the technical details, but the big picture, so I wanted to start with some simple questions, to help prime the discussion around API development.
- Where? - Where are APIs being deployed. On-premise, and in the clouds. Traditional website hosting, and even containerized and serverless API deployment.
- How? - What technologies are being used to deploy APIs? From using spreadsheets, document and file stores, or the central database. Also thinking smaller with microservices, containes, and serverless.
- Who? - Who will be doing the deployment? Of course, IT and developers groups will be leading the charge, but increasingly business users are leveraging new solutions to play a significant role in how APIs are deployed.
The Role Of API Definitions While not every deployment will be auto-generated using an API definition like OpenAPI, API definitions are increasingly playing a lead role as the contract that doesn’t just deploy an API, but sets the stage for API documentation, testing, monitoring, and a number of other stops along the API lifecycle. I want to make sure to point out in my API deployment research that API definitions aren’t just overlapping with deploying APIs, they are essential to connect API deployments with the rest of the API lifecycle.
Using Open Source Frameworks Early on in this research guide I am focusing on the most common way for developers to deploy an API, using an open source API framework. This is how I deploy my APIs, and there are an increasing number of open source API frameworks available out there, in a variety of programming languages. In this round I am taking the time to highlight at least six separate frameworks in the top programming languages where I am seeing sustained deployment of APIs using a framework. I don’t take a stance on any single API framework, but I do keep an eye on which ones are still active, and enjoying usag bey developers.
Deployment In The Cloud After frameworks, I am making sure to highlight some of the leading approaches to deploying APIs in the cloud, going beyond just a server and framework, and leveraging the next generation of API deployment service providers. I want to make sure that both developers and business users know that there are a growing number of service providers who are willing to assist with deployment, and with some of them, no coding is even necessary. While I still like hand-rolling my APIs using my peferred framework, when it comes to some simpler, more utility APIs, I prefer offloading the heavy lifting to a cloud service, and save me the time getting my hands dirty.
Essential Ingredients for Deployment Whether in the cloud, on-premise, or even on device and even the network, there are some essential ingredients to deploying APIs. In my API deployment guide I wanted to make sure and spend some time focusing on the essential ingredients every API provider will have to think about.
-Compute - The base ingredient for any API, providing the compute under the hood. Whether its baremetal, cloud instances, or serverless, you will need a consistent compute strategy to deploy APIs at any scale. -Storage - Next, I want to make sure my readers are thinking about a comprehensive storage strategy that spans all API operations, and hopefully multiple locations and providers. -DNS - Then I spend some time focusing on the frontline of API deployment–DNS. In todays online environment DNS is more than just addressing for APIs, it is also security. -Encryption - I also make sure encryption is baked in to all API deployment by default in both transit, and storage.
Some Of The Motivations Behind Deploying APIs In previous API deployment guides I usually just listed the services, tools, and other resources I had been aggregating as part of my monitoring of the API space. Slowly I have begun to organize these into a variety of buckets that help speak to many of the motivations I encounter when it comes to deploying APIs. While not a perfect way to look at API deployment, it helps me thinking about the many reasons people are deploying APIs, and craft a narrative, and provide a guide for others to follow, that is potentially aligned with their own motivations.
- Geographic - Thinking about the increasing pressure to deploy APIs in specific geographic regions, leveraging the expansion of the leading cloud providers.
- Virtualization - Considering the fact that not all APIs are meant for production and there is a lot to be learned when it comes to mocking and virtualizing APIs.
- Data - Looking at the simplest of Create, Read, Update, and Delete (CRUD) APIs, and how data is being made more accessible by deploying APIs.
- Database - Also looking at how APIs are beign deployed from relational, noSQL, and other data sources–providing the most common way for APIs to be deployed.
- Spreadsheet - I wanted to make sure and not overlook the ability to deploy APIs directly from a spreadsheet making APIs are within reach of business users.
- Search - Looking at how document and content stores are being indexed and made searchable, browsable, and accessible using APIs.
- Scraping - Another often overlooked way of deploying an API, from the scraped content of other sites–an approach that is alive and well.
- Proxy - Evolving beyond early gateways, using a proxy is still a valid way to deploy an API from existing services.
- Rogue - I also wanted to think more about some of the rogue API deployments I’ve seen out there, where passionate developers reverse engineer mobile apps to deploy a rogue API.
- Microservices - Microservices has provided an interesting motivation for deploying APIs–one that potentially can provide small, very useful and focused API deployments.
- Containers - One of the evolutions in compute that has helped drive the microservices conversation is the containerization of everything, something that compliments the world of APis very well.
- Serverless - Augmenting the microservices and container conversation, serverless is motivating many to think differently about how APIs are being deployed.
- Real Time - Thinking briefly about real time approaches to APIs, something I will be expanding on in future releases, and thinking more about HTTP/2 and evented approaches to API deployment.
- Devices - Considering how APis are beign deployed on device, when it comes to Internet of Things, industrial deployments, as well as even at the network level.
- Marketplaces - Thinking about the role API marketplaces like Mashape (now RapidAPI) play in the decision to deploy APIs, and how other cloud providers like AWS, Google, and Azure will play in this discussion.
- Webhooks - Thinking of API deployment as a two way street. Adding webhooks into the discussion and making sure we are thinking about how webhooks can alleviate the load on APIs, and push data and content to external locations.
- Orchestration - Considering the impact of continous integration and deployment on API deploy specifically, and looking at it through the lens of the API lifecycle.
I feel like API deployment is still all over the place. The mandate for API management was much better articulated by API service providers like Mashery, 3Scale, and Apigee. Nobody has taken the lead when it came to API deployment. Service providers like DreamFactory and Restlet have kicked ass when it comes to not just API management, but making sure API deployment was also part of the puzzle. Newer API service providers like Tyk are also pusing the envelope, but I still don’t have the number of API deployment providers I’d like, when it comes to referring my readers. It isn’t a coincidence that DreamFactory, Restlet, and Tyk are API Evangelist partners, it is because they have the services I want to be able to recommend to my readers.
This is the first time I have felt like my API deployment research has been in any sort of focus. I carved this layer of my research of my API management research some years ago, but I really couldn’t articulate it very well beyond just open source frameworks, and the emerging cloud service providers. After I publish this edition of my API deployment guide I’m going to spend some time in the 17 areas of my research listed above. All these areas are heavily focused on API deployment, but I also think they are all worth looking at individually, so that I can better understand where they also intersect with other areas like management, testing, monitoring, security, and other stops along the API lifecycle.
I have self-sensored stories about microservices, because I have felt the topic is as charged as linked data, REST, and some parts of the hypermedia discussion. Meaning there are many champions of the approach, who insist on telling me, and other folks how WRONG they are in the approach, as opposed to helping us work through our understanding of exactly what is microservices, and how to do well.
For me, when I come across tech layers that feel like this, they are something that is very tech saturated, with the usual cadre of tech bros leading the charge, often on behalf of a specific vendor, or specific set of vendor solutions. Even with this reality, I've read a number of very smart posts, and white papers on microservices in the last year, outlining various approaches to designing, engineering, and orchestrating your business, using the "microservices diet".
Much of what I read nails much of the technology in some very logical, and sensible ways--crafted by some people with mad skills when it comes to making sense of very large companies, and software ecosystems. Even with all of this lofty thinking, I'm seeing one common element missing in most of the microservices approaches I have digest--the human element.
I hear a lot of discussion about technical approaches to unwinding the bits and bytes of technical debt, but very little guidance for anyone on how to unwind the human aspect of all of this, acknowledging the years of business and political decisions that contributed to the technical debt. Its one thing to start breaking apart your databases, and systems, but its another thing to start decoupling how leadership invests in technology, the purchasing decisions that have been made, and the politics that surrounds these existing legacy systems.
I don't know about you, but every legacy system I've ever come across almost always has had a gatekeeper, an individual, or group of individuals who will fight you to the death to defend their budget, and what they know about tech (or do not know). I've encountered systems who have a dedicated budget, which only exists because the system is legacy, and with that gone, the money goes away too--sell me a microservices solution for this scenario!
Another dimension to this discussion, is that investors in microservice solutions are not interested in their money being used for this area. It just isn't sexy to be spending money on dealing with corporate politics, and unwinding years of piss poor business decisions, and educating and training the average business user around Internet technology. If you do not unwind these very human led, politically charged, business environments, you will never unwind the systems that exist within their tractor beams. Never. I'd care how much YOU believe.
In the end, I'm not trying to make you feel like you are going to fail. My goal is to encourage more investment in this area by the microservice pundits, vendors providing solutions the space, and VC's who are pouring money into these solutions. Many of the young, energetic folks at the helm of startups do not fully grasp the human side of corporate operations, and the potential quagmire that exists on the ground in front of them.
I am hoping that a handful of service providers out there can lower the rhetoric around their services and tooling, so that expectations get set at more realistic levels. Otherwise the push-back against the first couple failed waves of microservice implementations will become impenetrable, and blow any chance of making it work.
We closed the call for papers for APIStrat Austin 2015 last week, and have been processing the almost 100 talks that were submitted. We already had some session topics identified for the API conference this November 18th, 19th, and 20th in Austin, TX, but were also delighted to see the vast number of topics submitted through the call for papers process, resulting in this tag cloud.
This edition of APIStrat doesn't have a theme--we are just making about the roots of the conference, and looking to focus the conversation around the most pressing issues we face across the space at the end of 2015--I think this tag cloud reflects that conversation.
We have gone through all the talks, grouped them by topic, and will be reviewing them with the session chairs we have assigned to help organize and manage each track. By October, we will begin publishing the calendar, and sharing more detail about each session track, and the speakers in attendance.
While call for papers is closed, we have a handful of sponsorship opportunities left, so contact us if you want to help support the conversation at APIStrat. At the very least, make sure you are registered before the very intimate event sells out, you won't want to miss the conversations that occur.
Yeah, I know, it is a click-bait title. My goal is to not flame the haters, but educate everyone involved, both the people looking to learn about microservices, and those who are practicing microservices, so I'm hoping you will forgive me--my goal is not page views.
I'm sticking to my goal of not making microservices a regular thing I talk about, but I wanted to share some observations after listening to folks talk at Gluecon last week--a week that was full of microservices love (and hate). In the past, I mentioned that I received a handful of emails, and DMS from folks letting me know I'm not doing microservices, and after Gluecon I think I have a beginning list of why I think people generally are telling me I'm doing it wrong.
- Tooling - People tend to identify with the tooling they use, and deeply associate the function with the tool. So if I don't use Puppet or Chef, I'm not orchestrating. If your not using Jenkins, your not doing continous delivery, and so on. This is dangerous for microservices storytelling, but also is something that will continue.
- Vendor - This is probably the most egregious reason I see people telling others they are doing microservices wrong, and is somewhat akin to the tooling item, but is really just rooted in selling you something. You will never do it right, unless you are their customer, so don't worry about it.
- Scope - You just aren't doing it at the scope someone else is doing it. Microservices is something large enterprises do, not small companies, or individuals. You have to be operating across many availability zones, many servers, etc. Horseshit. Microservices can be done at any scope, don't listen to them.
- Completeness - You are only doing some of the aspects of microservices. Maybe you don't need to orchestrate across many operational zones, and your traffic looks very different than some of the usual suspects. Microservices is very seldom a complete picture, it is more about doing what works for you, cherry picking the valuable items you can apply in your world, not any complete definition.
- Incomplete Info - Someone just read a single blog post you wrote, and hasn't been following the series, asked any questions, and just make assumptions about a single thing you wrote. While I feel you should make each post as complete as possible, the responsibility is on the reader to do their homework - Bwahahaha! yeah right, pundits will never do that.
So far these are the biggest reasons that I've extracted from people me telling me I am not doing it right. Ultimately it comes down to power and control, which I saw play out with SOA, REST, Linked Data, and other areas of "API". They either want to make you feel small, or not cool because you aren't using the tools they are using, or they just want you to buy their snake oil.
In the end, for me, microservices are just a new storytelling package. They are mostly marketing, as many of the skeptics like to point out, and the companies who are truly doing microservices, will care about sharing their real-world stories, and work overtime to help you understand--there are a handful of them out there, like Runscope and Iron.io.
For me, I will be focusing on the moving parts of microservices, which is why you'll see me talking about API design, deployment, management, testing, monitoring, security over just "microservices" umbrella. I could easily attack many of the pundits who challenge me, and say you aren't doing API management, evangelism, monetization, or other critical areas right, but why would I? Where does that get us?
As usual, if you have any questions, let me know how I can help. I'm happy to share, as I try to make sense of all of this. I'm adding some new research sites to support the conversation.
I've been monitoring an emerging slice of the API, that has been dubbed "microservices" for some time now, and you've even heard me explore its use, when describing my architectural approach to redesigning more core API stack for my internal systems. I’m slowly redesigning my internal API stack, keeping them as small as possible, throughout all aspects of operations, deploying each endpoint in a single dockerized container, that contains the OS, database, web server and all server side API code. I'm also using Github, in conjunction with APIs.json to assist me in my orchestration, throughout every aspect of these APIs lifecycle from design to testing—all a new approach for me when managing my APIs.
Is defining my APIs in this way, so that the definition is much more than just the actual API endpoints, but also include the entire backend, and lifecycle workflow Microservices? Fuck if I know. One thing I do know, is I've had enough conversations with folks who are doing microservices, resulting in me saying several times that the micro services definition is something very personal. Something which the first 10 times I said , sounded very positive, but after experiencing multiple people telling me what I'm doing isn't microservices, I can tell the term definitely will continue to be very personal—very much in the same way conversations around REST or Hypermedia has gotten personal on forums, and HN threads.
You know what this tells me about microservices? Is it is more about power, control and influence, everything from enterprise architects stating their endorsement of the concept, down to the individual architects, who love to make sure you know that the way you are doing it, is WRONG! I don't think there is a single definition of "microservice", and much like REST, I think we'll hear plenty of “right” definitions, by the enterprise justifying what they are doing, and those who are selling something to the enterprise. Is there anything wrong with this? Nope. Just not my style, not my game, and has shown me over the couple months how the concept is not a fit for me.
I’m going to stick in the realm of API. It is a completely bullshit term, that means everything, but you know what? Somehow it escaped the IT, architect, and vendor ownership that SOA and now microservices possesses. For me APIs have long been more than just the tech, it is also about the business and politics of any platform implementation, so why can’t it contain my architecture styles as well? So from here forward I will be closely paying attention to microservices conversations, but you won’t hear me use the word, as I’m more comfortable in API land, and what it has come to mean to the wider public—I just don’t give a shit about what the enterprise adopts, or doesn't.
Microservices will reflect the power and control of the entities behind them, unlike APIs, which also possesses that characteristic, but also quickly shifts the conversation to be more about the innovation and opportunity that developers bring to the table, and the end-users who put that to use. Let’s make sure as we do not get caught up in the architectural discussions behind the technical curtain, and that we remember why all of this API stuff is working. Let's not get caught up in all the value being just about our architectural styles, and re-live a classic IT tale, ignoring that the real value is what people actually do with our services.
The Enterprise Will Make The Same Mistakes With API And Microservices That They Did With SOA, Because Essential API Concepts Go Right Over Their Head13 Mar 2015
I would say the enterprise space fleet has successfully shifted their course, heading in the general direction of everything API. SAP, Oracle, IBM, Microsoft, and the rest of the usual suspects have pledged their faith to APIs, mobile, and IoT, all in a very public way. For those of us who jumped off the SOA bandwagon a while back (getting on the API bandwagon, yeehaw), this can be amusing to watch, cause the API bandwagon is better y'know?. ;-)
I sincerely hope that these large companies can make a shift towards an API way of life, or a microservice devops way of operating, or whatever the cool enterprise kids are calling it these days. Some companies will find success in these areas, while many, many others will just talk about it, and buy a wide range of vendor services that will help them talk about it—there will be a lot of money to be made (and spent).
The thing I fear, is that many of the core principles of why this whole thing is working will go right over the heads of many technologies and business leaders in the enterprise. Elements like simplicity, transparency, openness, and functioning in a decoupled, decentralized way. You are the enterprise, many of the essential ingredients of API, are just fundamentally at odds with what you are. You are a big, complex, powerful, and political corporate presence—a beast that does not like being taken apart, reinvented over and over, and being forced to work in concert with other internal, partner, and public resources, in an open way.
This doesn't mean that some enterprise entities won’t be successful—some will. I will be carefully watching the waves of enterprise organizations, for these tech savvy few, who internalize the core philosophy around APIs, successfully uploading their corporate soul to the API singularity. However, I predict I will have to work very hard to find these companies amidst the wreckage of the enterprise organizations that make the same mistakes they did with SOA, because APIs are just not compatible with their DNA, and ultimately will be rejected for reasons that are more enterprise than they are API.
I'm just getting started exploring the ways to use APIs.json when it comes organize my new Docker fueled, micro services stack. I’ve been using APIs.son to describe each micro service, as well as define the overall collection of almost 20 micro services. I’m using the include collection, as a navigation for the loosely coupled stack of micro-services, and my friend Chris Spiliotopoulos (@chefarchitect), has done some more work with his Swagger.ed, delivering some APIs.json goodness that is in alignment with where I want to take all of this.
Adding to the discovery and visualization work he did for Swagger, Chris enabled Swagger.ed to look for valid APIs.son files as well, so when you browse to any APIs.son, like the one I have for my micro services stack, you see the little APIs.son search icon in the address bar.
When you click on the APIs.son search icon in the address bar, you get a very cool visualization. Its pretty basic at the moment, just a visual catalog of the APIs available in the include collection of my stack, but when you connect up with the Swagger visualization work he's already done, we could have a pretty cool API catalog for managing and exploring microservices.
I have almost 20 micro-services listed, and Swagger.ed gives me the ability to navigate it in a very interactive way. Whats next? We don’t know…it is about exploration, and finding out the most meaningful way of exploring the APIs I deploy and aggregate into APIs.son collections.
I can see having a visual catalog of all of my API design that I collect, then deploy, and evolve them as needed for various parts of my infrastructure or for my clients, into other sub-collections? Disconnected collections? Loosely coupled collections? Not sure how tight I want things, part of the micro service definition for me is the size of the network connecting the services, as well as the services theselves.
API Management Infrastructure And Service Composition Is Key To Orchestration With Microservices In A Containerized World02 Feb 2015
As I work to redefine my world using microservices, I have this sudden realization how important my API management infrastructure is to all of this. Each one of my microservices are little APIs that do one thing, and do it well, relying on my API management infrastructure to know who should be accessing, and exactly how much of the resource they should have access to.
My note API shouldn’t have to know anything about my users, it is just trained to ask my API management infrastructure, if each user has proper credentials to accessing the resource, and what the service composition will allow them to do with it (aka read, write, how much, etc.) My note API does what it does best, store notes, and relies on my API management layer to do what it does best--manage access to the microservice.
This approach to API management has llowed me to deploy any number of microservices, using my API management infrastructure to compose my various service packages—this is called service composition. I employ 3Scale infrastructure for all my API / microservice management, which I use to define different service tiers like retail, wholesale, internal, and other service specific groupings. When users sign up for API access, I add them to one of the service tiers, and my API service composition layer handles the rest.
Modern API management service composition is the magic hand-waiving in my microservice orchestration, and without it, it would be much more work for me to compose using microservices in this containerized API world that is unfolding.
Disclosure: 3Scale is an API Evangelist partner.
I'm using APIs.json to organize my swagger defined microservices running in docker containers, and using the machine readable API index to drive navigation between microservices organized in a single collection. APIs.json provides a simple, machine readable way to index the technology, business, and political elements of each microservice I deploy.
As I was auditing the 18+ microservices I’ve setup for my core operations, I wanted to audit each one, and make sure I had included various elements in each APIs definition, like where developers can onboard, terms of service, and where to find related code samples. To accomplish this, I was able to just compare each microservice APIs.json, with a master APIs.json template I had established while planning my original microservice stack.
Sometimes it is just as important to know what is missing, right alongside what is available for all of my microservices, and APIs.json is proving to be a simple way of understanding this layer of my API operation at scale.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.