stplanr | Robin Lovelace

Aggregating lines, part II

Thu, 14 Feb 2019 00:00:00 +0000

The previous post demonstrated a new method to aggregate overlapping lines. It showed how to combine 2 lines that have an area of overlap. More excitingly, it led to the creation of a new function in stplanr, overline_sf(), that lives in the development version of the package. The purpose of this post is to provide an update on the status of the work to refactor the overline() function, in a human friendly alternative to discussion in the relevant GitHub issue: https://github.com/ropensci/stplanr/issues/273

Set-up

To re-cap, these are the package versions we’ll be using in this post (run this line to get the development version of stplanr if you want to reproduce the results):

remotes::install_github("ropensci/stplanr", ref = "refactor-overline")

The library is loaded and the input data from the previous post was loaded as follows:

library(stplanr)
routes_fast_sf$value = 1
sl = routes_fast_sf[2:3, ]
sl$value = c(2, 5)
overline_sf2 = function(sl, attrib) {
 attrib = "value"
 attrib1 = paste0(attrib, ".1")
 sl_intersection = sf::st_intersection(sl[1, attrib], sl[2, attrib])
 sl_intersection[[attrib]] = sl_intersection[[attrib]] + sl_intersection[[attrib1]]
 sl_intersection[[attrib1]] = NULL
 sl_seg = sf::st_difference(sl[attrib], sf::st_geometry(sl_intersection))
 rnet = rbind(sl_intersection, sl_seg)
 return(rnet)
}

This was used as the basis of a demonstration of how overline works, demonstrated in the following code chunk and resulting plot:

rnet = overline_sf(sl, "value")
plot(rnet["value"], lwd = rnet$value)
plot(sl["value"], lwd = sl$value, col = sf::sf.colors(2, alpha = 0.5))

Dealing with many lines

The above method worked with 2 lines but how can it be used to process many lines? Clearly the same function could be implemented on another line, but it would need to work from the 3 lines of the newly created rnet object rather than the original 2 routes. Let’s introduce a 3^rd^ route into the equation, that does not intersect with this newly created rnet object:

sl3 = routes_fast_sf[4, ]
rnet = overline_sf2(rnet, sl)
plot(rnet$geometry, lwd = rnet$value)
plot(sl3, col = "red", add = TRUE)

In this case the method of adding to rnet is simple: just add the entirety of the line to the rnet object:

attrib = "value"
rnet3 = rbind(rnet, sl3[attrib])
plot(rnet3$geometry, lwd = rnet3$value)

This works fine. In fact it works better than the original overline function because it does not add the value of the existing thickest line in the previous figure onto the new line, a problem associated with overline.sp() that is illustrated in the following code chunk and resulting figure:

sl1_3 = as(routes_fast_sf[2:4, ], "Spatial")
rnet3_sp = overline(sl1_3, attrib = "value")
plot(rnet3_sp, lwd = rnet3_sp$value)

A question that arises from the previous example is this: what if the next line intersects with the route network? It is no longer possible to simply add together two values. This can be illustrated by introducing 2 more lines:

sl4_5 = routes_fast_sf[5:6, ]
plot(rnet3$geometry, lwd = rnet3$value)
plot(sl4_5$geometry, col = "red", add = TRUE)

Both the new lines intersect with the newest part of the route network. This means that we cannot simply rbind() them to it as we did for sl3. They need to be dealt with separately.

Before we deal with them, it’s worth taking some time to consider what we mean by ‘intersect’. Intersection is actuall a specific type of geometric relation between 2 sets of features. We can see the type of relation by using the function st_relate():

relations = sf::st_relate(sl4_5, rnet3)
relations
unique(as.vector(relations))

This shows us something important: although 2 elements (1 and 4) of rnet relate in some way to the new lines, only the 4^th^ feature has a linear, overlapping relation with it. That relation is 1F1F00102 which, as far as I can tell, is not a named spatial predicate (FF1F00102 means ‘intersects and touches’ but does not have a linear overlap). This relation is what we need to decide whether or not to simply bind a new feature to the growing rnet, whether we need to break it up (or at least part of it) into smaller lines before doing so (it also raises the wider question of which order should we do things).

In the simple case of whether to simply bind the next line (4) onto rnet3 the answer is simple now we know the string code associated with linear overlaps. First we’ll test it on the previous example of sl3 and the original rnet composed of 3 features:

relate_rnet_3 = sf::st_relate(rnet, sl3, pattern = "1F1F00102")
relate_rnet_3
any(lengths(relate_rnet_3))

The FALSE meant there was no linear overlaps. So we simply used rbind(). When we ask the same question of rnet3 and sl4, however, the answer is TRUE:

sl4 = sl4_5[1, ]
relate_rnet_4 = sf::st_relate(rnet3, sl4, pattern = "1F1F00102")
relate_rnet_4
any(lengths(relate_rnet_4))

How to proceed? We need to avoid rnet objects containing any overlapping lines. Because sl4 overlaps with part of rnet3 we will need to remove the overlapping line, run the overline_sf2() function, and then re-combine the result with the pre-existing route network object. We can split-up the rnet3 object into overlapping and non-overlapping features as follows:

sel_overlaps = lengths(relate_rnet_4) > 0
rnet_overlaps = rnet3[sel_overlaps, ]
rnet3_tmp = rnet3[!sel_overlaps, ]

We can check that there is only one overlapping feature as follows:

nrow(rnet_overlaps)

And we can proceed to join the two features together using our new function:

rnet_overlaps4 = overline_sf2(rbind(rnet_overlaps, sl4[attrib]))

Adding this back to the rnet3 object results in an larger rnet object incorporating all the value and geometry column data we have so far:

rnet = rbind(rnet3_tmp, rnet_overlaps4)
plot(rnet$geometry, lwd = rnet$value)

The information provided so far informed the creation of the following function:

stplanr::overline_sf

This function made use of the slightly simpler individual route-joining function, and associated helper functions:

stplanr::overline_sf2

stplanr::overlaps
stplanr::to_linestring

Aside from being clunky, this approach has the additional flaw of not working. It works fine for lines 2 to 6 in routes_fast_sf, as illustrated below.

r6 = routes_fast_sf[2:6, ]
rnet6 = overline_sf(r6, attrib = "value")
plot(rnet6)

A problem arises when we try to run the same command on the same data, but with one more line added. That will be the topic of the next post on aggregating lines to form route networks, which will also introduce a new function overline2(), developed by my colleague Malcolm Morgan.

Another benchmark

To provide a taster of the new function, let’s see it in action in a benchmark on this latest route network consisting of the five lines illustrated in the previous plot. Again, we use bench::mark() to benchmark each approach:

bench::mark(check = FALSE,
 overline = stplanr::overline(r6, "value"),
 overline_sf = stplanr::overline_sf(r6, "value"),
 overline2 = stplanr::overline2(r6, "value")
)

Again the results are not Earth-shattering and should be taken with a large pinch of salt: performance should not be evaluated against these relatively tiny datasets, but against the scale of datasets that the functions were designed to handle: city scale route networks involving thousands of overlapping lines. The same principle is used in the benchmarking section of the h2oai website: focus on benchmarks that apply to the datasets you will actually be using. In this case, my hypothesis is that the geometric functions overline() and overline_sf() will perform disproportionately badly as the size of the input dataset grows (to be tested). Still, it is interesting to note that the new overline2() function uses way more memory than the other functions. More next time!

Aggregating lines, Part I

Sat, 27 Oct 2018 00:00:00 +0000

Introduction

It’s been a busy 12 months but with the Geocomputation with R book nearing completion¹ I’ve finally found some time to update my blog and do a bit of thinking, about the tangled topic of line aggregation.

Why aggregate lines?

The transport ‘flow’ on any particular segment of the transport networks is the aggregate (sum) of trips that pass through it (Hollander 2016). Finding the flow across a transport network based on input data composed of individual routes, is therefore an aggregation problem. It requires a more complex solution than that provided by the aggregate() function in the base R (R Core Team 2018) package stats, however, because the geometry of the output LINESTRINGs will be fundamentally different than the input LINESTRINGS of the routes: a route network is composed of many small way segments, but a route is a single long LINESTRING.

Aggregating lines with overline() MKI

Creating such a route network, with aggregated values per segment, is the problem that the overline() function in the R package stplanr was designed to solve (see a 3.5 yr-old question on gis.stackexchange.com for more context).

The function works well and is the basis of the Route Network layer (MSOA) in the Propensity to Cycle Tool (PCT). In fact it was developed precisely for this purpose, as illustrated in the image below, which shows a common visualization/analysis problem encountered by transport researchers when working with multiple routes: overlapping routes are not easy to identify from non-overlapping routes: notice the red lines in the centre of Leeds in the image look the same as the red lines on the outskirts, despite representing much more movement.

To overcome the problem overline() was born (credit to Barry Rowlingson who wrote the foundations of the function), and it works (on a tiny dataset of 2 lines) as follows:

library(stplanr)
sl_sp = routes_fast[2:3, ]
sl_sp@data = data.frame(value = c(2, 5))
plot(sl_sp, lwd = 9, col = "yellow")
rnet_sp = overline(sl_sp, attr = "value")
plot(rnet_sp, add = TRUE, lwd = rnet_sp$value)

The utility of such a function is illustrated in the figure below, which shows the original route network layer of the PCT in action over a similar area of Leeds:

This works great and the resulting network is used for strategic network planning: you can download route network data in the ‘Region data’ tab of the PCT (Lovelace et al. 2017). The route network data for Leeds can, for example, be downloaded as a .geojson file from here.

But there are some issues: the function works on the older SpatialLinesDataFrame class defined in the sp package (Pebesma 2018). This data class has been superseded by the simpler sf class in the sf package, which is faster than sp for some (if not many) operations. Another issue with overline() is that in some cases when 2 lines meet, the resulting aggregated line can be longer than it should be. So there are performance and functionality issues to address. Rather than solve them all here, this post sets-out the issue using reproducible code and suggests next steps for a new overline() function (or perhaps just an updated overline.sf() function which currently just wraps overline.sp()).

Aggregating lines with sf

Rather that starting from scratch and writing a geographic algorithm from the ground-up, we will start by exploring solutions provided by existing packages, notably sf, which provides an interface to the GEOS library.

Let’s start simple, with just 2 lines, which have an associated amount of flow (with illustrative values of 2 and 5 in this case):

library(stplanr)
sl = routes_fast_sf[2:3, ]
sl$value = c(2, 5)

These lines clearly have a decent amount of overlap, which can be extracted using the function st_intersection():

sl_intersection = sf::st_intersection(sl[1, ], sl[2, ])
plot(sl$geometry, lwd = 9, col = sf::sf.colors(2, alpha = 0.5))
plot(sl_intersection, add = TRUE)

Furthermore, we can find the aggregated value associated with this new segment as follows:

sl_intersection$value = sum(sl_intersection$value, sl_intersection$value.1)

We still do not have a full route network composed of 3 non-overlapping lines, however: the original lines need to be ‘clipped’ so that they do not overlap with sl_intersection. This can be done as follows:

sl_seg1 = sf::st_difference(sl[1, ], sl_intersection)
sl_seg2 = sf::st_difference(sl[2, ], sl_intersection)
plot(sl$geometry, lwd = 9, col = sf::sf.colors(2, alpha = 0.5))
plot(sl_seg1, add = TRUE)
plot(sl_seg2, add = TRUE)

We now have all the geographic components needed for a route network. The only remaining task is to combine them, using rbind, right? Not quite: the following command fails:

rnet = rbind(sl_seg1, sl_seg2, sl_intersection)

Lesson: we need to be more careful in isolating the value to aggregate. We will therefore run the previous stages again, but with attrib set to the attribute we would like to aggregate over (value in this case):

attrib = "value"
attrib1 = paste0(attrib, ".1")
sl_intersection = sf::st_intersection(sl[1, attrib], sl[2, attrib])
sl_intersection[[attrib]] = sl_intersection[[attrib]] + sl_intersection[[attrib1]]
sl_intersection[[attrib1]] = NULL

That leaves us with a ‘clean’ object that only has a value (7) for the attribute column name we want (value).

On this basis we can proceed to create the other segments, keeping only the column we’re interested in. To save time and typing, we’ll create both segments in a single command:

sl_seg = sf::st_difference(sl[attrib], sf::st_geometry(sl_intersection))
rnet = rbind(sl_intersection, sl_seg)

It worked! Now we’re in a position to plot the resulting route network, with ‘width’ proportional to the flow along each segment:

plot(rnet, lwd = rnet$value)

A benchmark

To test that the method is fast, or is at least not slower than the original overline() function, at least for this task, we’ll package-up the method in a new function:

overline_sf2 = function(sl, attrib) {
 attrib = "value"
 attrib1 = paste0(attrib, ".1")
 sl_intersection = sf::st_intersection(sl[1, attrib], sl[2, attrib])
 sl_intersection[[attrib]] = sl_intersection[[attrib]] + sl_intersection[[attrib1]]
 sl_intersection[[attrib1]] = NULL
 sl_seg = sf::st_difference(sl[attrib], sf::st_geometry(sl_intersection))
 rnet = rbind(sl_intersection, sl_seg)
 return(rnet)
}

If you are new to scripts/algorithms/functions, it may be worth taking a look at the new Algorithms chapter in our near-complete book that teaches a range of geographic methods that use R (Lovelace, Nowosad, and Muenchow 2019). Now the method has been put in a function, we can compare its performance with the per-existing overline() function for comparison:

bench::mark(check = F,
 overline.sp = overline(sl_sp, attrib = "value"),
 overline.sf = overline(sl, attrib = "value"),
 overline_sf2 = overline_sf2(sl, attrib = "value")
 )

The results are not Earth-shattering: the new function seems to be around the same speed as the original, if a little faster. This is great news, but remember: the new function only works on 2 lines so is much simpler. More work needed!

Next steps

The next step is to generalist this method so it works for many (potentially thousands) of lines in a way that scales, something that should help on the visualization side, a topic that attracts much interest (see for example this gis.stackexchange post on the subject and, more broadly, a recent article showing how to make animated maps by Jakub Nowosad). There is also work to do on performance but, as Donald Knuth said (Knuth 1974):

premature optimization is the root of all evil (or at least most of it) in programming

So a more complete overline.sf() function is needed. That will (hopefully) be the topic of the next blog post.

References

Hollander, Yaron. 2016. Transport Modelling for a Complete Beginner. CTthink!

Knuth, Donald E. 1974. “Computer Programming As an Art.” Commun. ACM 17 (12): 667–73. https://doi.org/10.1145/361604.361612.

Lovelace, Robin, Anna Goodman, Rachel Aldred, Nikolai Berkoff, Ali Abbas, and James Woodcock. 2017. “The Propensity to Cycle Tool: An Open Source Online System for Sustainable Transport Planning.” Journal of Transport and Land Use 10 (1). https://doi.org/10.5198/jtlu.2016.862.

Lovelace, Robin, Jakub Nowosad, and Jannes Muenchow. 2019. Geocomputation with R. CRC Press.

Pebesma, Edzer. 2018. “Simple Features for R: Standardized Support for Spatial Vector Data.” The R Journal.

R Core Team. 2018. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

Any input on that still welcome - see the contributing section for more. ↩︎