Eyes on Earth Episode 130 – NLCD Accuracy

[00:04] Hello, everyone, and welcome to another episode of Eyes on Earth, a podcast produced at the USGS Eros Center. Our podcast focuses on our ever-changing planet and on the people here at Eros and across the globe who use remote sensing to monitor and study the health of Earth. My name is Tom Adamson. [00:30] is a land cover product produced at Eros, and it's widely used for land cover and change research in the U.S. [00:37] Eros is also responsible for making sure it's as accurate as it can be. [00:42] In her work on the reference and validation team for annual NLCD at Eros, land remote sensing scientist Joe Horton samples thousands of 30-meter Landsat pixels to identify the land cover class those pixels should be. [00:58] Why do this? [00:59] to provide a reference dataset for NLCD users so they know about the accuracy of each of the NLCD Land Cover classes. In this episode, we'll learn a lot more about how that works, and even take a close look at a sample pixel to see how the team validates these pixels. [01:20] Joe, will you go ahead and introduce yourself? Certainly. My name is Joe Horton. [01:25] I am a land remote sensing scientist at the USGS Aero Center, working as a KBR contractor, and I am currently the technical lead for the reference and validation team for the annual NLCD product. You mentioned annual NLCD. That's a new release that just came out.

[01:42] October 24. But first of all, tell us really quick [01:45] about NLCD. What is the National Land Cover Database? That's actually a land cover and land cover change product that the USGS has been putting out for quite some time. The original release would have been a map from 2001. Prior to annual NLCD, there were nine what they call epochs, so time phased [02:05] releases. So 2001, 2006, just as Snackshots in Time. That maps land cover across, we'll be talking about CONUS today, so the lower 48. The conterminous United States. Yes. Or connected 48 states. There's lots of different ways to say it. Yeah, the lower 48 or the conterminous U.S. or the, I'll call it CONUS for simplicity. It basically maps the land cover at what's called a [02:29] a level two level of detail which means instead of mapping just forest per se it divides it into subclasses of deciduous forest evergreen forest and mixed forest the annual nlcd which is the product we just released is a continuation of that project taking it to the next level basically instead of mapping at the epochs like the legacy nlcd does we're mapping those land cover and land [02:59] through the current release which is 2023. And then in the future we'll release 2024, 2025. So it'll be land cover across [03:07] uh CONUS every year which is like the next step in NLCD.

[03:13] What is your role in NLCD? I work on the what's called the reference and validation or the RNV. [03:19] section of the project. Basically, we are collecting the data that will be used to test and check the accuracy of the annual NLCD land cover and land cover change products. In a high level nutshell, we are collecting data [03:36] If you picture the world as 30 meters by 30 meters, a Landsat pixel, about the size of a baseball diamond, we are collecting the land cover and land change information at up to 10,000 of those locations across CONUS for every year. So basically my team, we get a plot, we look at it, we look at the Landsat data, we look at high resolution imagery, and we go through and we say every year, [04:01] from 1984 through 2023, what was on that plot? What was the land cover? And then if there was a change, when was it? And what kind was it? And then those labels that we're assigning for every year are going to be [04:15] compared to those products at that same location, same year, to give a metric for how accurate the products themselves are. Okay, sounds good. How many plots are you looking at to validate? It's going to be up to 10,000. We've already collected a little over 5,000. It's a two-phase collection. So phase one was the first 5,000. Now we're on to the second group, which could be up to 5,000, the final number still being determined. [04:40] Are these pixels that you're looking at completely random? Is there a reason that any of them are chosen? The first 5,000, that's why it's phase one. Oh, okay. Phase one was completely random.

[04:50] So it was 5,000 pixels chosen across CONUS, just completely random. We just basically bounded it to what are we mapping? [04:57] So, you know, we don't go way out into the ocean, for example, and then just pick 5,000. Phase two is a what's called a stratified random sample, which means once we had the products, the products that were released in October, we could use that to create bounding boxes for various situations, various classes and stratify the second set of samples. For example, to make sure we have enough samples. [05:24] locations, [05:25] in rare classes. [05:27] Because as you can imagine, with a random sample, [05:30] rare things on the landscape are going to be rare in the random sample and you need a certain number [05:37] of samples in order to even be able to do any sort of [05:41] robust statistical calculation. If you only have 12 out of 5,000 in a class, it's not going to be statistically rigorous. - Not good enough. - Right. - Not rigorous enough. What is one of those rare classes? [05:54] One of the rare ones that we ran into, and people kind of jump when I say this because you don't think it's rare until you think. Keep in mind, we're talking the entire lower 48. [06:06] the entire surface of the lower 48, [06:09] high intensity developed. [06:11] I am surprised. So those areas that are basically... [06:15] 100% cement or building. Yeah, yeah. Middle of a city, downtown. Right. Parking ramps. Right. Roof of a shopping mall. Yeah. You know, those kind of things. That's actually rare. It is.

[06:26] 48 statewide. - Exactly. They're the part that jumps out people think of because [06:32] we think of [06:33] Atlanta, we think of New York City, we think of LA, and they are [06:37] Relative to other cities, they are huge. But relative to the Rocky Mountains as a whole, [06:45] you know, or the Great Plains as a whole. [06:48] They're a minuscule part. It's a small percentage. Exactly. So when I when I gave you the example of 18, that was [06:56] the in the random sample that was approximately how many high intensity develops we got out of those 5 000. so that's that's a good example okay so the next phase in validation we'll be looking at [07:06] few more of those right for example right working on without going into all of the statistics involved in how you determine strata and a stratified random sample there are lots of papers out there or statistical textbooks that can go into all of that but it's not fun radio [07:23] Basically, we use a selection method that allows us to somewhat [07:30] target to those classes that we know we didn't get enough of. Or at least areas we think have a higher probability of having those classes. We're trying to get to, you know, 200, 225, something like that. [07:44] samples per class to allow for some statistical rigor. All right. That's not something that I thought of. Yeah. That you would have to think about. Those are the fun kind of things that I get to deal with and think about is how many samples do we need and how do we get them?

[07:57] How many people are doing this work? How many people are on the team? Well, obviously the annual NLCD team is larger. If you're talking the RNV team and you're talking the people who are doing [08:07] the you know boots on the ground nuts and bolts interpretation type work the data collection we have five interpreters that are doing the you know those up to 10 000 plots looking at them every year there's also another there's also a group of people who are what i call my qa qc my quality checkers who review [08:29] some subset of plots that get [08:31] sent to them for review for a variety of reasons. And then I've got people who are helping with, you know, data creation and management and all of that. So there's about eight of us when it's all said and done, if you take it down to those nuts and bolts people. Okay. [08:46] What is it like when you're examining one of these pixels? Ah, that's when I get to put on what I call my Sherlock Holmes from Space hat. [08:53] Our primary data collection tool, our primary evidence is the Landsat data. [08:57] and we use a tool called TimeSync [09:00] to visualize all of that data. It basically lets us see [09:04] the Landsat imagery for that 1984 through 2023. Basically every image, assuming it's not completely cloud covered, is available to us to view that and we can use a variety of different band combinations. So like most people think of photographs and they're you know the true color that's what they're familiar with. We actually can harness the additional information that Landsat has from like the near infrared and the shortwave infrared and those kind of things which are better at visualizing some

[09:33] particular aspects of vegetation or fire regimes, things like that. So we have the Landsat data. We also have what we call the trajectory, which is taking all of those [09:44] land side images and mapping a value to our plot location for each of them. So you can see [09:51] how a plot changes [09:53] throughout a growing season, throughout a year, so you kind of get the feel for what does that plot do [10:00] Normally. [10:01] You know how in the winter deciduous trees are just bare. [10:05] And then you can watch them and they start to green up and they start to green up and then boom, they're green and they're green and they're green and then they fall, they start to change their colors and the leaves get red and then they fall off. [10:14] And of course we can see that in the Landsat imagery that happens throughout the year. Yeah, and that's what the trajectory values let me see. I get a feel for that. So I... And you're moving your hand up and down as you talk, which is kind of fun because we can't see that. [10:25] It's those changes is what you're indicating in the reflectance for that. Exactly. Yeah, I can see. Okay, green up and down and green up and down. I'll look at the plot. I'll look at the trajectory and the first thing I look for [10:38] is does that wave [10:40] change somewhere. Okay. Yeah. It might have a natural rhythm like trees. [10:45] but you're looking for a difference. I'm looking, yeah, that's, that's one of the first things I look for is does anything jump out at me as there's, [10:52] you know, that rhythm broke. [10:54] Then what I can do is using the Landsat imagery and using high resolution imagery. Think Google Earth. That's something most people are familiar with. Those kind of images where I can look back and start to narrow down

[11:07] what's on the landscape, and if something changed, what happened and when. And then we record that for every year. So in my case, I usually have two, sometimes three computer monitors with a variety of information up that I and all the other interpreters, we are [11:24] our specialty is basically synthesizing all of that information down so we can [11:29] put nature in a box, put a label. In 1996, this plot was Evergreen Forest, and move on from there and do that for every plot. [11:39] Every year. Well, that's what's interesting here is you're not just looking at one... [11:43] 30 meter pixel. - No. - You're looking at that 30 meter pixel [11:47] back through time. Yes. Looking all the way back to 1984 and then looking up to 2023. And you're also not just looking at an image. You're looking at [11:55] the scientific measurements that Lionsat gave you. Yep, all sorts of different, they call them indices. You know, you can look at different... [12:03] calculations of those different bands we were talking about that can tell you different things about what was happening on the landscape. And then putting that all together and saying for this [12:14] plot for this [12:15] 40 years basically Here's the story. Yeah, here's what happened and I I'm biased, but I think it's awesome It is really cool. I like the idea that every pixel has a story. Yes, that's really interesting. I [12:28] way to look at it. So you mentioned this time sync tool. Are we able to kind of walk through and get a glimpse of what you do in that tool? Sure if you want. I mean I'll try to describe. We'll have to describe what you're doing but we'll give it a try. Here just a second.

[12:42] All right, so like here is an example of one of the plots or what we would see when we look at a plot. Let's describe what you're seeing across the top there. There's a graph with a [12:53] Lot of dots. A lot of dots. And you can see on the bottom it's got the years. [12:58] So 85, 86, 87. - And across the bottom looks like [13:02] Well, it looks a little pixelated and blurry because you're really, really zoomed in. I am. Yeah. But these are the landsat images. These are the landsat images. And yeah, they look, they look blurry both because I'm zoomed in and also because [13:14] As a general rule, as human beings, we're used to [13:18] what we see in a photograph. [13:19] Keep in mind, each one of those pixels is the size of a baseball diamond. It's gonna blur compared to what you're used to looking at. [13:28] So yeah, so we've got the trajectory, that's all these dots, and that's where, when you were talking before, where I had the wave going, you can start to see [13:36] the peaks in the growing season. So looking at this plot you can see, you know, 1984 through somewhere around 1997, it's all pretty [13:45] The values are all pretty similar. Yeah, it's not like the same thing. [13:48] throughout those years, there's some variation, but it looks like a kind of rhythmic, there's a pattern. [13:54] Yeah. Over those years. Yeah. [13:56] Vegetation, [13:58] has a pulse, you know, because because even even evergreen vegetation [14:03] calms down a little in the winter. I mean, it's still it's still photosynthetic, but it's, you know, it's got a shorter day to deal with. You're not going to have the exact same value every single day. But yeah, you can kind of see where it's a range that's

[14:17] pretty consistent and a pulse that you know a wave that's pretty consistent up to about 1997 and then you can see here suddenly change yeah suddenly the values [14:29] drop, [14:30] pretty dramatically. TimeSync lets us, like I said, we can look at different band combinations, it also lets us look at a variety of different calculations. [14:38] different indices. So we can look at wetness indices. We can look at burn [14:46] We can look at all sorts of different things. [14:50] Each one has its use and things that it's better at. What I'm showing here is one that's really good for showing what happened here, which is forest harvest. [14:58] oh okay logging is going on yes okay um so yeah it had the the values were all pretty similar and then this dramatic drop [15:05] which if we looked at the [15:07] Thank you. [15:07] Landsat imagery in the in the trajectory you can see it's happened somewhere around 1998 1999 and [15:14] part of what we have to do as interpreters is nail down the exactly when. [15:18] And if we looked in, if we looked at all of the chips for 1998, which you can see I'm showing them here, this window here is showing all of the all of the Landsat images. [15:30] from 1998 and you can see there's what about 21 of them [15:34] Bye. [15:35] looks like. Yeah. [15:37] And we can use that to say, okay, [15:40] Here, that plot was still pretty green. [15:44] Tone wise, [15:45] here's where it starts to turn

[15:48] brown, which is an indicator in this band combination and this indices of less vegetation. [15:54] So we know that the change occurred in mid to late 1998, which is one of the things we as interpreters have to... [16:03] make sure we verify what year did that change occur? Because we're trying to map change every year. [16:10] And you know that this wasn't a natural change because of this graph that you were looking at first. Well, there's a couple of bits of information that go into it. You can see a graph similar to this for a forest fire too. [16:22] So the trajectory lets me know [16:25] Seeing that drop in the values lets me know a change occurred. Figuring out what the change was... [16:31] is a synthesis of the available evidence. Okay, good. So... That's why you're Sherlock Holmes. Exactly. So what I would look at in this kind of situation, the things that I [16:42] that point to it being a harvest even just looking at Landsat. We haven't even gotten into the high resolution imagery yet. [16:48] but just looking at the Landsat imagery, [16:50] is the fact that it's a very [16:53] geometrically shaped [16:55] disturbance. We make the call based on what happens at that pixel, but we do use the context around it to have to figure out what the what [17:04] "It's." So it's very rectangular and things like fire [17:09] don't tend to be geometric. I say tend to because of course there are things like prescribed burns which [17:16] Get bounded. [17:17] The fact that it's

[17:19] Geometric makes me think human cost. [17:22] versus a natural thing like a flood or a landslide or a fight. Then what I can do is look at the high-resolution imagery, which I actually have a synthesis of. So like at this point, [17:34] You're not ready to say this is definitely logging. At this point, I would be confident to say [17:41] that this [17:42] Disturbance occurred in 1998 and that it was something human caused. Okay. [17:47] but for this for our recording we do want to to the best of our ability say what the disturbance type was. Was it a harvest? Was it [17:56] a fire, was it whatever. [17:58] So if I look at the high resolution imagery, like we talked about what people would see in like Google Earth. [18:04] That is what then helps me further narrow it down. In looking at the high resolution imagery from 1993, I can see it was a forest. I already knew that because I recognize what a forest looks like in the Landsat imagery. But to double confirm. And then looking at from 1999, [18:21] which is after that event we were just talking about, [18:25] In the high resolution imagery, you can see it was completely clear cut. There is no tree left. If it had been a fire, the signal... [18:34] in the trajectory would have looked slightly different, but there'd also you normally you would see debris. [18:41] Oh, I see. You might see some downed [18:44] trees, down trees or still standing dead ones or, you know, not always because,

[18:49] Sometimes they go in and they salvage log or whatever. But... [18:53] with what I was seeing in the landsat imagery, what I was seeing in the trajectory values, and what I'm seeing in the high resolution imagery, now I would be comfortable saying in 1998, this plot, [19:03] was a clear cut. [19:05] And then if we go back, then the next thing is, okay, well, that's great. Now I know what happened, what it was from 1997 to 1997. I know it was forest. [19:13] I know it was harvested and clear cut in 1998, [19:16] But now I got to call the rest of it, too. Yeah, you got to bring it up to date now. Right. So going back to the, you know, if I was flipping through the high res, high resolution imagery, I know it goes back to trees eventually, and I actually [19:29] Also, kind of cheat looking at this image. You can see they're in lines. So I also know it was planted. It's kind of like a tree farm or something going on here. Or at least some sort of managed forestry. [19:41] And if you look, going back to our lovely trajectory, so we had that drop in 1998. And then you see how it's kind of got this slope going up. And how it's kind of changed in color a little. It goes from really bright red to kind of, I don't know, pea soup green. And then eventually it gets back up and it's... [19:59] a fairly similar slightly different but similar value range to what we were seeing before the harvest and you've got that clear [20:06] wave of annual pulse again. Basically, this plot was clear cut, [20:12] and took a couple of years for, you know, anything other than weeds and grass and, you know, that kind of stuff to grow in. Some small,

[20:20] woody vegetation starts growing in. [20:23] We don't consider it a tree until it's more than five meters tall. So there would have been a period of what we would call low woody or shrub until that woody vegetation gets to that minimum height, in which case it returns to trees. And this plot has been... [20:38] evergreen forest ever since. [20:40] Where did this TimeSync tool come from? Who developed it? TimeSync was developed [20:47] quite some time ago. It was originally the Forest Service and Oregon State University. In around 2010 they originally developed it. The Forest Service was using it for one of their [20:58] projects. [21:00] LCMAP, which was a precursor to the project I worked on before annual NLCD, [21:05] was we collaborated with the forest service to collect reference data. So that's how we were introduced to the time sync tool. [21:12] When annual NLCD came along, [21:15] we wanted to continue using that tool so we had to [21:19] bring a version in-house to customize it. [21:22] because the version that was originally used by developed by the Forest Service [21:27] You remember back when I talked about level two and deciduous, evergreen and mixed? Okay. It did not have level two [21:34] level of detail. It just had forest. Forest is forest. Right. So we had to [21:39] make sure it could collect the level of data we needed to go to this level two classification. Clearly you can't do this to [21:47] all of the billions of pixels that cover conus i could but it would take a long long long long time

[21:55] - [21:56] just thinking that there must be some automation. And what you're doing, does what you're doing inform the automation that happens to identify all those billions of pixels as their land cover types? [22:06] Yeah. [22:07] There is a lot of [22:08] machine learning and automations and algorithms involved in the creation of the annual NLCD products, the classification, putting a label on each of those billions of pixels like you talked about. [22:20] what i am doing right now with the reference and validation [22:25] There is some automation in things like helping us select those 5000 random ones from all those billions of pixels. But what I do right now, a lot of it is actually manual interpretation. [22:39] part of [22:40] having the ability to have [22:43] automations or you know, [22:45] AI, machine learning. [22:48] is that you need to have [22:50] high quality [22:51] robust and a lot of [22:54] Data. [22:55] to train the AI on. [22:59] And [22:59] There's not a lot [23:02] of data sets like the one we're creating here, the reference and validation. [23:06] So there's not a lot of [23:08] data to train an AI on yet. [23:11] part of [23:12] the appeal and the interest that I constantly am hearing from like when I went to AGU and present it. [23:19] there is a lot of interest in the reference and validation data itself because there's not a lot of data sets like that out there.

[23:28] because it's a lot of manual and time intensive and [23:33] you need people with either the ability to learn or the specialized skills to do what I just showed you, which is not intuitive to a lot of people. [23:42] Our data has to be collected independent of anything labeled on [23:48] the product maps. Otherwise, you know, if we looked at the map products and said, well, what does it call this point? [23:56] we'd be teaching to the test, which would [23:59] bias the accuracy assessment. Can you talk more about what the reference data set is important for? Are there uses outside of this accuracy assessment? [24:10] - Potentially, yes. And based on the number of questions I got at AGU, there's people who want it. So obviously the primary driver for the collection right now [24:23] is [24:24] to provide accuracy metrics for the annual NLCD products. That's the primary driver right now, to get that data, to compare our labels, [24:33] to what the map [24:34] algorithm labeled those locations and give the users [24:39] a metric for how accurate is this product. The NLCD users. Yes. Exactly. So they can see for the [24:47] year or years they're using for the classes they care about, what is the accuracy. As far as other uses for the reference data, [24:55] its potential, [24:56] for people to, assuming that their label

[24:59] their labels, their land cover definitions align with what was used when we collected the data. [25:06] I could potentially see someone using it to test their own types of classification algorithms. There's a lot of research going into different types of machine learning algorithms for classification of land cover and land cover change, or using our data for some sort of, you know, other training or testing. We're also doing something kind of unique with our collection in that we talked about [25:28] each interpreter collecting at those plots. 50% of our plot locations have another [25:33] second interpreter look at them independently. And then we can compare those labels and look at, you know, [25:41] do the interpreters agree on what is at that location on those years? And there's been... [25:48] several papers over the last couple of years looking at how does [25:51] interpreter consistency and interpreter agreement in a collection like this [25:57] feed into overall [26:00] accuracy metrics and area calculations and things like that. [26:03] Do you know if this type of accuracy assessment that you do is unique to NLCD? There's a lot of work going into different [26:11] land cover maps right now. Everything from small, you know, area mapping to state, country, [26:18] even people attempting to do global maps. And [26:23] The gold standard of accuracy assessments for those types of products is to do something similar to what we're doing. Collect an independent,

[26:33] air quotes here, ground truth label at those locations and use that to calculate the accuracy of your product. There are projects out there doing... [26:45] doing this, but there are also [26:47] a lot of projects that do [26:49] other versions of validation things like using [26:54] a subset of their training data, they set that aside. They don't use it to train their algorithm. They compare that to get their accuracy metric. [27:01] The fact that we're collecting it from 1984 through 2023 and across all of CONUS, I don't know that anybody else is doing anything at that level. There are definitely other people out there who do photo interpretation. But to do it... [27:16] across that length of a time scale and at that Anderson, at the level two level of detail, there's not a lot. [27:26] of that being done out there. [27:27] It is the gold standard of how you validate a land cover product, but [27:32] It's tough to do. That's why it's gold. Yes. If you're going to create a map you should tell people how accurate it is. [27:39] So creating the reference data set, doing the validation is what will allow the users to be able to look at those products and know, yes, [27:48] this this is a product I want to use. It is it works for my area of interest. It covers the land cover I want. Also kind of on a more philosophical side, knowing [28:00] the accuracy [28:01] And [28:02] Bye. [28:02] extension, the inaccuracies,

[28:06] is what allows us to move land cover mapping science forward. [28:10] You know, if we know where [28:12] the current algorithm struggles. Are we, I know regions where there are [28:18] particular challenges. That's what lets us then work to [28:23] move the science forward and address those problems. If you don't know there's a problem, [28:29] you can't fix it. [28:30] What's one of the biggest challenges in doing this work? The challenge is... [28:35] coming up with [28:37] protocols and definitions [28:39] that are clear, [28:41] and can be consistently applied [28:44] across multiple interpreters and across time. We need protocols that basically would let five different people look at this location and come to the same conclusion. And not just the current five. [28:56] In the future, if you get a new interpreter comes in or someone retires, that knowledge needs to be transportable. And it's not [29:03] as simple and clear cut as people tend to think. Well, a tree is a tree, isn't it? Yeah, no. That is my classic example. I spend a lot of time discussing [29:16] What makes a tree? [29:18] And then how many trees [29:20] does it take to make a forest? Because [29:23] "What is a tree?" [29:24] if you ask a botanist an arborist [29:28] a forester and a homeowner, [29:30] you are gonna get four different answers. [29:34] for what is a tree.

[29:36] I can't have four different answers for what is a tree. [29:39] Luckily, I love it. It's a puzzle to me to figure out how can you define something, especially when nature don't like to be put in buckets. Nature doesn't like to be stamped with a label. Nature likes to throw curveballs. [29:54] figuring out those labels and those protocols, making sure that they're as clear [30:00] and unambiguous as we can, [30:03] is a challenge. [30:05] But it's also a lot of fun. Oh, that's cool. Well, maybe this is the same question. What what's your favorite thing about the work that you do? [30:14] My favorite thing is [30:16] The story. [30:17] being able [30:18] to look... [30:20] at a unique perspective of the world [30:23] and figure out, [30:24] "What happened here?" And knowing, figuring out that story, you know, for all of these different locations. I've looked at tens of thousands of locations at this point, and I still find things that I'm like, I have never seen that before. [30:36] It gives you a unique perspective on the world. It's kind of like I got the overview effect, but in miniature. [30:42] That feeling that astronauts talk about when they are up in space and they see the Earth, and it just changes the way you view the world because you realize how big and diverse it is. I get that, but I just sit at my desk. [31:00] Thank you to Jo Horton for talking with us about her work on reference and validation for annual NLCD. And thank you, listeners. Check out our social media accounts to watch for all future episodes.

[31:12] also subscribe to us on Apple and YouTube podcasts. This podcast, this podcast, this podcast, this podcast, this podcast is a product of the U.S. Geological Survey Department of Interior. [31:37] you [31:38] you [31:39] .