17. Integrating Terraform with Heroku

Owen Ou: Yeah, gotcha. It's basically Infrastructure as Code for across all the platforms.

Mars Hall: Exactly. Yeah, and Infrastructure as Code. I'm glad you said that. That's, IaC is a very popular acronym for this. Basically, it means that instead of logging into the web console for Amazon Web Services, and clicking and creating all your separate resources, and having to figure out how it all fits together, and then maybe having to reproduce it again some other day. With Infrastructure as Code, instead you actually describe your infrastructure, and you can check it in like a source code to, say, git. Then, you use Terraform to execute this code, and it will create all of these things for you.

Mars Hall: You can do it reproducibly, and that's the beautiful thing about Infrastructure as Code, is that it lets you treat your infrastructure like you do software. Where you can test it, and you can experiment wit h different ways of putting it together, and easily destroy it and recreate again.

Owen Ou: This is our open source project so everybody can make your software for free, I guess.

Mars Hall: Yeah, yeah. HashiCorp does have an enterprise premium version that you pay for, and it is a pretty cool product. You can use Terraform fully without paying for it. Really, the main thing is that when you run just the free version of Terraform, you're responsible for running Terraform, like, on your own computer, or on a server of your own. Whereas, with HashiCorp's enterprise product they actually run it for you. It's a great way to get into it, is to use the free version. Yeah.

Owen Ou: How does Terraform compare to other tools like Chef, Ansible or CloudFormation?

Mars Hall: I mean, each of those are, you know, they definitely allow you to describe Infrastructure as Code. Chef is a much older technology, and if you're unfamiliar with Chef it basically lets you create these recipes in Ruby language that you can execute, and have it create bootstrap servers. The things is, is that it can be really hard to reason about what Chef configurations do, because it's not a declarative, simplified configuration format. It's a full blown programming language.

Owen Ou: Yeah, and that requires the operator to learn Ruby, so if anyone doesn't have Ruby experience people would have to learn Ruby just to use Chef.

Mars Hall: Yeah, indeed. Even for a Ruby expert it can be really challenging to understand what a Chef config does. So, that's Chef. Then, like with CloudFormation, that's an AWS specific tool. Like, that's Amazon's own way to templatize infrastructure. Of course, a big advantage of Terraform is that you can use it with many different providers at the same time.

Mars Hall: Yeah, really when you think about it, Terraform is Infrastructure as Code, so when you look at Ansible, for instance, that's actually a tool that's about essentially provisioning the state of your servers. When you think about Terraform it's usually more like, "How do I put the right network pieces in place? How do I put the right servers in place?" Terraform doesn't answer how you bootstrap the server, and load your app onto it. Well, for most providers.

Mars Hall: For Heroku, we do provide that, because that's the premise of the platform, but yeah.

Owen Ou: Yeah, we will definitely be getting to the Terraform Heroku plug-in in a bit. What do you think are the use case for Terraform?

Mars Hall: Really, in my own work, I find it a super powerful tool. Instead of using a provider directly, whether it's Heroku or AWS, I find it so much simpler to essentially formulate the structure of the pieces that I need for a project as code, and just apply that with Terraform. It makes it so much easier to experiment with the structure of your infrastructure as you iterate and go through cycles of understanding how these things work together, and how they work in the long term. You can refine your configurations.

Mars Hall: Really, if you're willing to learn it, which it is, it's really graceful to learn. It's not super complicated. I find it's a great neutralizer for making it easy to experiment with platforms, and understand how the pieces can fit together. Whether you're building a small project, or you're part of a company, this is a great example where it's, like, at many startups, whoever puts the infrastructure together first mends up the servers. It becomes this big hodgepodge of things that can never be reproduced at some point.

Mars Hall: If you begin a project as a startup, if you just start from the get-go by provisioning things with Terraform, and keeping your configurations and source control, then there's this tractability of, like, anybody that joins the company can understand, like, what is the infrastructure, and how can I change it, and contribute to it. Whereas, when you don't use Terraform or something like it, it can be really hard for new people that come to your team to understand, like, what is all the infrastructure, and how do I change it?

Owen Ou: Yeah, definitely. A wise way for you to automate your infrastructure, and also more like provide a spec for a newcomer to learn about how your application are deploying.

Mars Hall: Yeah, totally.

Owen Ou: Yeah, let's get into the community a little bit of Terraform. You mentioned, Terraform is an open source project. What is the community like?

Mars Hall: Terraform itself, Core as they call it, it is open source. The primary contributors are actual HashiCorp engineers on the Terraform team. They are open to pull requests, so really, anybody can walk up if you're familiar with Go. All of this is written in the Go programming language. You can walk up and open a pull request to fix a bug, or to add a feature. My own personal experience is that the HashiCorp team is very busy, and so it can take time to get feedback on things you want to contribute.

Mars Hall: Then, when you step outside of Terraform Core, so the premise of Terraform is that there's this Core engine that actually does the work to apply configurations and whatnot. Then, there are plugins, and the plugins are called providers. The providers are things like the Heroku provider, or the AWS provider, or DNSimple provider. Each of those are open source Go language projects, which are usually... So, the community around each of those is really based on that platform.

Mars Hall: Like, for Heroku, our provider, the community there are Herokai, like myself and a few engineers from Salesforce. I would say that we've got a really responsive, friendly community around our Terraform provider. Then, different ones, like if you were to approach the AWS provider, for example, it is a huge provider. AWS is huge, and there's a lot of people that contribute to it. That's another place where it's like, the community is really based on who is actively working on it, and who's paying attention to GitHub, and answering pull requests and that kind of stuff.

Owen Ou: Let's say I want to fix up a Terraform Heroku plugin. How should I get started?

Mars Hall: Yeah. Obviously, you need a GitHub account.

Owen Ou: Everybody has a GitHub account now.

Mars Hall: Yeah. Yeah, if you don't have a GitHub account, go ahead and sign up. Take that step. The first advice would be, just look at the issues we have open in GitHub on our current provider. There's a good chance that what you're interested in is already being talked about, or if you found a bug there's a good chance we already know about it, and it's accounted for there. Really, this is just good for any open source. Before you open an issue, go look at the issues and see, like, are other people working on this?

Mars Hall: That being said, you can pretty much clone down the repo, follow the instructions. We have a full acceptance test suite on it. To run that, you have to have a Heroku account that you essentially plug your API key into the acceptance tests. Then, the Terraform provider actually tests itself by creating things in your Heroku account and verifying they're correct, and then destroying them.

Mars Hall: It's pretty simple to work through solving, adding features, and then you open a pull request and we'll review it. The HashiCorp folks themselves will sometimes review pull requests, because all the official providers are actually owned by HashiCorp. They're actually at the Terraform providers GitHub org. Yeah, so it's pretty easy to contribute. Definitely, before you spend time, like, blood, sweat and tears, like, engineering a solution, open an issue and just check with us. Like, what do you think about this? Are you already focusing on something like this? We'll give you feedback, definitely.

Owen Ou: Yeah, sounds good. What would be a good channel to reach out to you guys? I mean, is there a Slack channel, or email would be preferred, or what would you prefer?

Mars Hall: I mean, the GitHub issues are the way. Yeah, for sure. You can obviously probably track some of us down in other ways, but if you want to talk about the Terraform provider, definitely open an issue. Even if it's not literally a bug, like, if you just want to open an issue that's, "Proposal, I want to add this kind of resource, or this kind of feature," that's a great way to communicate with us.

Owen Ou: Now that we talk about the Heroku Terraform plugin, how does it work with the Heroku platform? How does it interact with it? Let's say I want to create an app. What is the flow like?

Mars Hall: Yeah, yeah. Every Terraform provider is really just an API client for that business service. In our case it just talks to our GA platform API. I say GA, the Generally Available one. That's what's at api.heroku.com. It uses your API key, so you have to set an API key. We do have a great Dev Center article at the Heroku Dev Center called, Using Terraform with Heroku. That gives a great overview for how to set up the provider.

Mars Hall: Then, in the case of, say, creating an app, like, you might not need to Terraform just an app, but it's a great way to get a feel for how you can basically templatize your Heroku architecture, right? Basically, you create a main.tf Terraform file on your computer, and you declare that you want to use the Heroku provider. Probably with a version, and then essentially you define a Heroku app, and you give it a name and a region. Then, you save that file, and Terraform apply.

Mars Hall: Terraform will say, "Hey, this is what I'm about to do to your infrastructure." You want to review what it says, make sure you're not going to be destroying anything. Which, if you're just creating an app you wouldn't be destroying anything. Then, you type in yes, and Terraform begins its work. Talking to the API to essentially make the resources in your account match what you have specified in your configuration.

Mars Hall: Something I should've mentioned earlier, it's really, honestly, one of the most beautiful things about Terraform is that you can think of Terraform as a parallel execution tool that, when you declare this configuration of resources, some of the resources might depend on other resources. If you have a Heroku app and you want to attach a Postgres database to it, then you specify a Heroku app resource, and then a Heroku add-on resource.

Mars Hall: Terraform understands because of the way that the provider is structured, that it has to create the app before it can create the add-on.

Owen Ou: Ah, I see, so Terraform internally will build a dependency graph of all the resources that constitute your deployment.

Mars Hall: Exactly.

Owen Ou: Be that an app, an add-on, or maybe some other resources like S3. That kind of thing.

Mars Hall: Yeah, exactly. It's really cool, because as you get to more sophisticated architectures, what ends up happening is that you might have some resources. Like, at Heroku we have private spaces. Those take about eight minutes to provision. We have Kafka Clusters, and those can take 8-10 minutes to provision. If you have all these different resources, and they're not explicitly dependent, or inferred dependent on each other, then Terraform can do those two things in parallel. It can start provisioning a private space, and start provisioning a Kafka Cluster, and do that in parallel.

Mars Hall: It's a really powerful tool. You can also explicitly label resources as dependent on other resources so that, in some cases there's dependencies that are softer, that aren't explicit in the kind of way the provider is structured. A good example of that is an app. If you provision an app, you might have another resource that uses that app somehow. A great example is that you could create a Heroku app that contains an API gateway. That API gateway, if you use a technology like maybe the Kong API gateway, it itself has a Terraform provider that you can configure it with.

Mars Hall: You might end up in a situation where you need to have Terraform create the Kong app, but then you don't want it to do any of the Kong configuration until you know that API gateway has become alive, and available. You can essentially create an explicit dependency with a condition where it will wait until it gets a healthy response back from that app before it proceeds with doing more things.

Owen Ou: I see, so some of the available, only available app runtime. For example that you mentioned, requires an app to exist, and probably talk to an app URL?

Mars Hall: Yeah. Exactly, exactly. This is actually one of the really cool things about using Terraform with Heroku. For a lot of other providers, you're really just creating infrastructure. You still have to use a tool, whether it's something like Ansible, which there are Ansible integrations for Terraform provisioners, as they call them. In order to actually make something run on your instance, like, deploy an app to it. With something like Heroku, because we have literally building and running the app is a fundamental part of our platform you can actually, yes, provision fully running apps directly with Terraform.

Owen Ou: How does Terraform help with multi-app systems? Because, at runtime team, we run into the problem all the time, so we have the same app that need to configure for a different environment. For example, we have EU, we have US East, US West. Does Terraform help in this case?

Mars Hall: Yeah, so it definitely can. One of the ways that it could help is if you actually want to provision the same system architecture multiple times into different regions. In that case you might have a configuration that, so all configurations take input variables. Those input variables can come from options on the command line, or it can come from environment variables. They can also come from external service that use data sources that you query within your configuration.

Mars Hall: The gist is that you can have, you know, like, pass in a variable, which is which region to deploy to, or to provision this in. For US East you have that region set, and then for whatever, US West you set that region differently, and you actually provision multiple instances of that configuration with different variables to put it in different places.

Owen Ou: We probably forgot to talk about the language used behind Terraform.

Mars Hall: Ah, yes.

Owen Ou: It's a HashiCorp Configuration Language, HCL. Tell me more about it.

Mars Hall: First of all, I mentioned already that Terraform and the providers are written in Go. As a user of Terraform, you don't actually ever touch Go unless you want to contribute to Terraform. If you just want to use Terraform you're not going to have to use Go. You'll use HCL. Basically, it's intended to be a declarative language. What this means is that, for instance, it doesn't matter what order you put things in a Terraform configuration. You can put, you can spread them across files, you can glom it all together in a single file. It doesn't actually matter, because what this language is designed to do is to specify this declarative set of resources that you want Terraform to figure out how to provision for you.

Mars Hall: HashiCorp Config Lang is very approachable in an English way. It's very friendly to having comments. It's very friendly as far as formatting goes. It's not whitespace sensitive. You can pretty much lay it out however you want, but it does, in what I think a beautiful modern trend is around, it has a Terraform fmt, format command, so that you don't really even have to think about how you write your syntax, because there's a standardized way that it will reformat the syntax for you.

Owen Ou: Yeah. That's similar to gofmt. Probably borrowed the idea from the Go language.

Mars Hall: Exactly. Also with JavaScript, like, Prettier, that kind of stuff.

Owen Ou: Nice. Well, why don't we switch track a little bit to talk about some of the best practices of using Terraform?

Mars Hall: Oh, definitely. There's a number of things. Some of the best practices really span whether you're using Heroku or not. Some of them are very Heroku specific. The first one I'm going to mention is probably the most important of all. In fact, I would elevate it out of best practices to biggest warnings. It's around what we call config drift. What that refers to is that you specify what you want your infrastructure to be configuration like in your Terraform config. You apply it, but then what if someone logs into your Heroku web console, or uses the CLI and changes something outside of Terraform? They just talk to the platform directly?

Mars Hall: Then, the next time you try to apply a Terraform config, you will experience config drift. Which means that the configuration that Terraform expects is no longer true. It's not longer in place. If you have simple infrastructure it can be pretty easy to get out of this kind of state. If you're on a team of people, and you have dozens, or hundreds of resources that Terraform is provisioning, you can get into a situation where it's really hard to make Terraform work again without doing dangerous things to try to make Terraform match what the new state of the world is.

Mars Hall: The advice here, after that whole story, is to express that when you use Terraform with Heroku, or with AWS, or any other provider, you need to take precautions that you don't change the resources directly with the provider, like through their web console, and cause config drift. There are some ways to do that in our Dev Center article about using Terraform with Heroku. We definitely guide through setting up a separate Heroku team to essentially contain everything that Terraform does for you so that you and the people that you work with on your team can know, well this team, everything in it is managed by Terraform. We only go through Terraform to change that stuff.

Owen Ou: Yeah, so rule number one, never change stuff outside of Terraform. I like it. What is the next best practices? I'm looking at the best practices page right now. The second one is avoid renaming Terraform apps.

Mars Hall: Oh, yeah. Every since Heroku came about, like what, 10 years ago now? We have strived to be very, very friendly as far as, when you're using our platform, we want to make everything feel as natural and simple as possible. From the get-go we've really focused on allowing you to address resources, like your apps, by their name. This is a name that you give them. It can be up to, I believe, 30 characters long. It's like, this name becomes, when you have a Heroku app, it's like, myapp.herokuapp.com. You literally end up getting a secure URL that includes this name in it.

Mars Hall: Because of this, there is actually, even outside of Terraform, if you rename a Heroku app that's part of a greater system that is in some ways using that app name as an identifier, obviously you're going to be breaking your system by renaming the Heroku app. This is just a best practice when you're using Terraform, because Terraform adds another level of trying to track the state and these IDs, so you want to avoid renaming your Heroku apps. You're better off destroying them with Terraform, and the recreating them with a new name.

Owen Ou: What if I couldn't destroy the app? Is there a manual way I can fix it?

Mars Hall: Yeah, so Terraform's really good at giving escape hatches for things like this. The Terraform command line offers a number of state functions, so you can do Terraform state show. You can actually remove things from Terraform State. You can also, depending on how you're storing your state, like, if you're just using the default file system, State Store, which is where, when you run the Terraform command it writes .tf State files right there in the directory where you're running the command. That's what actually tracks all the IDs and relationships of the resources.

Mars Hall: Anyway, you can also end up just editing that State file directly to search and replace. Like, the old app name, and putting the new app name in it, and that can frequently get you over this hump. We really don't want to advise people to manually edit Terraform State if possible, because it's just dangerous. You can end up doing something to your infrastructure that you didn't mean to, you know, as an accident.

Owen Ou: Gotcha, so it's tied back to rule number one. Always use Terraform to change your resource.

Mars Hall: If you're using Terraform, just use Terraform. Yeah.

Owen Ou: There's a third best practices. Use consistent name prefixes.

Mars Hall: Ah, yes. Once again, this is something that, whether you're talking about AWS resources, GCP, Heroku, one of the things that can be really confusing is if you have a configuration that has names of resources. Not just apps, but really anything that it creates. Then, if you login to the web dashboard and you look at what is in your account, it can be really hard to understand where these things came from. By declaring an input variable for your Terraform config that's the prefix you want to use, and then for all of the resources that you create in that config you just code into the configuration that you want it to start with, it's prefix.

Mars Hall: Then, when you list your Heroku apps, you can easily see, like, "Oh, this came from that Terraform configuration." Yeah, it also has a secondary effect that it means if you apply the same configuration with different input variables, that those different sets of resources that are created can be easily distinguished from each other as well.

Owen Ou: The next one is a little bit Heroku-specific. It's that, use one Heroku team per Terraform config.

Mars Hall: Oh, yeah. Basically, talked about this with the first best practice. About config drift. Essentially, one of the more challenging things is if you're programmatically provisioning these resources, and you need to be careful of not messing with them so that Terraform continues to work. Yeah, one of the best things you can do is just say, "Okay, let's name a team Mystartup-Terraform, maybe is the name of the team. Then, anyone that is looking at the Heroku dashboard on your team can easily see, like, understand, oh, okay. Everything in this team is actually managed by Terraform. Obviously, you can do these things different ways, but these will just help you be most effective with Terraform.

Owen Ou: The next best practices is use provisioner health checks.

Mars Hall: Oh, yeah. That's a cool one. You should really check out the Dev Center article to see the code that's sampled there. The gist of it is, like when I was mentioning that use case where you provision an API gateway as an app, and you need to know that that app is ready before you can send configuration to it. A provisioner health check, so Terraform provides these, they call them provisioners. There are different types of provisioners, but the idea is that it's an ad hoc script that can be run against that resource after it has been created, but before Terraform considers it created.

Mars Hall: What that specific one is talking about is, basically, for a given Heroku app, well you actually usually put this kind of thing maybe on the formation, which designates the actual dyno containers that run your app, but you specify a provisioner that basically is just a loop that uses the curl, like HTTP command line tool to just hit the, like, maybe the root, or maybe there's a health check endpoint on the app. It basically just keeps requesting that app until it gets a successful response. Then, it can say, "Okay, successful." Then, it lets Terraform proceed, knowing with confidence that that app is there, and running.

Mars Hall: It can be a great way to ensure that, even if you're not doing something like that where you have an API gateway and you need to know it's up for Terraform to do more work, provisioner health checks are also just great because it will mean that when Terraform apply completes, that you actually have, you know that all of the elements are actually up and running.

Owen Ou: Yeah, that's a pretty nice trick to make sure the app is healthy before proceeding with other resource provisioning. I can see that, like, you can write script to warm up you JVM, that sort of thing as well.

Mars Hall: Exactly. Yeah.

Owen Ou: All right. Great, so if you want to learn more about all the best practices of using Terraform, or Terraform, the Heroku plugin, feel free to check out our Dev Center article. Last questions, what does the road map look like for Terraform, or Terraform, the Heroku plugin?

Mars Hall: It's really exciting. Terraform has been working toward a really big release. Currently, they're at version 0.11, and they've been working toward version 0.12. One of the biggest changes with Terraform is that they're actually releasing an entirely new version of HCL, of the configuration language. It's really exciting, because while it's not purely backwards compatible, because of the way HCL is this very special domain-specific, declarative language, it had a lot of limitations in its original version.

Mars Hall: This new version of HCL is going to make things that sound pretty fundamental, but things like conditional statements, and loops, and using variables, and interpolating values of different kinds, like, it's all going to feel much more like a first class language. As opposed to being this kind of special configuration language. That's a really exciting part of what's coming with Terraform 0.12.

Mars Hall: Another thing is that, at Heroku we contributed a new backend State store to Terraform, which uses PostgreSQL.

Owen Ou: Excellent. I like this.

Mars Hall: If you're familiar with Postgres, Heroku has a amazing hosted Postgres offering, Heroku Postgres. In all of my work with Terraform, I was experiencing this challenge over and over again where it's like, either I store the State locally on my computer, and no one can collaborate with me on it, or I store it in the cloud, and the most typical way is using an S3 bucket and a DynamoDB table to manage locking. That is really awkward to set up, every single time.

Owen Ou: Yeah, so you basically set up two things just to make sure AWS can store a State, and with locking.

Mars Hall: Right, exactly. Especially if you're using Heroku, that means you're having to do a separate AWS thing, which while you can Terraform it, it's actually, it would have to be a separate Terraform configuration. Because, it's the thing that basically bootstraps Terraform being able to run.

Mars Hall: Anyway, realize that there is a real opportunity to make Postgres serve as a remote state store. That has been in the two betas, and the release candidate of Terraform 0.12. Once that's released you'll be able to use Heroku Postgres as your State store, which is really nice.

Owen Ou: Yeah, pretty excited about it. Maybe dip a tiny little bit of the detail. Does it use the Postgres database lock to lock the State and stuff like that?

Mars Hall: Oh, you're getting into it. Yeah, so actually it took two pull requests to get this through, because we realized there was an issue with locking. When we talk about locking, by the way, Terraform, when you are applying a configuration, Terraform has to keep track of all of these remote resource IDs, and how they relate to each other by writing this State file, which is a blob of data. If you're working on the local filesystem, like, no big deal. It's just you. As soon as you put it in a remote place, and you have a team of people there's a chance that several of you could Terraform apply at the same time. Then, what happens? Well, that's what the locking is about, right?

Mars Hall: Basically, Terraform has this kind of application level locking that, if two people both try to Terraform apply, only one of them will win. Basically, I'm not sure what all the behavior is around if it waits, or if it just errors out, but the point is that it prevents you from having multiple people conflict. The way we implemented that with Postgres is using locking. We use what are called session advisory locks. What this means is that it's just basically an integer ID lock that we have a specific way of picking which integer is the right one for the lock you're trying to take.

Mars Hall: Basically, when you Terraform apply, you'll get a lock, which is a session advisory lock, and you'll be able to operate. If someone else comes in, obviously that lock's already in place, and it will fail, not allow them to proceed. Then, essentially when your changes, when the apply is complete, Terraform finishes storing the State and unlocks.

Mars Hall: Now, one of the coolest things about using Postgres this way is that unlike other locking mechanisms that you'll find with the other remote state stores, you'll never get into a state where you have to force unlock it. If you're using the S3 backend with the DynamoDB locking, if the client that has it locked goes away, like, aborts, doesn't finish cleanly, disconnects from the network, that lock is just outstanding. Anyone, to make changes, then someone has to force unlock it. Which, of course, is kind of scary, because it's like, how do you know that that's not validly locked?

Mars Hall: With Postgres, because of the way it's sensitive to the client remaining connected, if, when you're using the Postgres backend, if your client drops away, the lock will actually be released automatically by Postgres.

Owen Ou: Yeah. Great, great. Definitely a great improvement against foul locking.

Mars Hall: Yeah.

Owen Ou: Yeah, because I heard story about, like, someone pushed an unfinished Terraform changes, and the other person has to manually unlock it.

Mars Hall: Yeah, yeah. Once again, it's like, the whole point of this Infrastructure is Code is really all about developing more trust in your infrastructure, so there's not a fear of changing it. That is one of the classic issues, right? Once again, you have people set systems up, and then everyone becomes scared to touch them, because you don't know what's going to happen if you change something.

Mars Hall: Terraform really helps you build that kind of team trust around knowing, like, this is what is configured, this is what's there, I can experiment and try applying this configuration in my own environment, see how it works before I actually roll it out to production.

Owen Ou: What about the Heroku provider? Anything new on the road map?

Mars Hall: Oh, on the Heroku provider. Well, yeah. I don't think I can really talk about exactly what's coming up in our API, but we do have a few things that are coming out later this year, in the next few months that will actually help us resolve some outstanding issues that, if you look at the issues in our Heroku Terraform provider you'll see that a number of them are tagged as Heroku API support. We're coming up to a point where we'll be able to implement a few things that have been desired.

Mars Hall: Other than that, we're really just interested in continuing to spread the word about it, see how people use it and, yeah, get feedback. We haven't implemented the whole of the Heroku API in Terraform. We've implemented the things that we have needed, and that other people have needed. If there are pieces that come up that you're like, "I can't believe this thing is not available," definitely, please just, once again, open an issue, request it or talk to us about how you might want to implement it.

Owen Ou: If I start a new Heroku project today, should I go with Terraform?

Mars Hall: Well, I would definitely say, make sure you're comfortable with Terraform first. If you're getting started, both with Heroku and with Terraform at the same time, it may feel a bit challenging, because there's so much new-ness. I also find that Terraform is fantastic for actually helping to understand new providers that I haven't used before.

Mars Hall: My favorite example is AWS, because there're so many things it does. Like, if you want to figure out how to do something with a given resource you have to search online, try to find docs. Well, if you just look at the Terraform provider, it's like this catalog of all the interfaces, all the things you can create. It definitely gives you this advantage to being able to have this coherency around what's available as far as resources go, and how you can just put them together, kind of like Lego blocks right there in your Terraform config.

Mars Hall: I personally just start almost everything with Terraform these days, because as I was saying earlier, if you just start a project with it then your infrastructure is code. Just like the app you're writing, and it can evolve and iterate over time. If you're comfortable with the technology, or even if you're not fully comfortable, but if you're starting to feel buy into it, I would say definitely. Just get started with Terraform right out the gate.

Owen Ou: Thank you very much for joining us today, Mars. Thank you for sharing with us Terraform, and Terraform Heroku provider.

Mars Hall: Awesome. Thank you, Owen, for hosting this fantastic conversation, and look forward to hearing from y'all on the Internet.