Programming Thoughts
J2EE - Pagination
Detail Page

Raison d'être

Search results pages display a limited amount of data and users want to see more in a details page, which needs to be lazy loaded.

Query Strategies

Attaching lazy loaded details would seem to be another implementation of the PageExtensionAssembler but applied to just the selected item. However, experience shows detail pages tend to have lots of details and it's easier to retrieve the record with all details attached, rather than attach every detail to an existing record. Further, the query Strategy can be used to re-read a database record after it's updated as other fields can be updated by back-end functions.

/** * Strategy for finding single item. */ @FunctionalInterface public interface SingleItemFinder<K extends Serializable,T extends Serializable> extends Serializable { /** * Retrieves item from primary key or id, or null if not found. */ public T getItem(K key) throws Exception; }

An example of a list finder is shown below.

public static class GameItemFinder implements SingleItemFinder { @Override public GameDTO getItem(Integer key) throws Exception { GameTable gameTable; GameDTO result; gameTable = Model.getInstance().getGameTable(); result = gameTable.find(key, GameOption.ARTISTS, GameOption.DESIGNERS, GameOption.MECHANICS, GameOption.PUBLISHERS); return result; } }

Existing query Strategies

Though unusual, list and page finders can read details as well, so the list cache need not, especially if the search results are small. Whether a finder should retrieve details can only be answered through experience. The changes to such finders are shown below.

@FunctionalInterface public interface ListFinder<T extends Serializable> extends Serializable { /** * Returns list, sorted if needed, of Data Transfer Objects. Null may be returned if nothing found. */ public List<T> getList() throws Exception; /** * Returns true if finder eager loads any optional details of each item. This is usually false as loading * details results in poor performance or there are no optional details. */ default public boolean getLoadsDetails() { return false; } } @FunctionalInterface public interface PageExtensionAssembler<T extends Serializable> extends Serializable { /** * Attaches additional data to each record for a requested page that the base record list finder did not load. */ public void assemblePageExtensions(Collection<T> items) throws Exception; /** * Returns true if finder eager loads any optional detail extensions of each item. This is usually false as loading * extensions results in poor performance or there are no optional extensions. */ default public boolean getLoadsDetails() { return false; } } @FunctionalInterface public interface PageByIdsFinder<K extends Serializable,T extends Serializable> extends Serializable { /** * Returns items from their ids. If any records are missing, the successfully retrieved ones must still be * returned. No item should ever be null. */ public Collection<T> getItems(Collection<K> keys) throws Exception; /** * Returns true if finder eager loads any optional details of each item. This is usually false as loading * details results in poor performance or there are no optional details. */ default public boolean getLoadsDetails() { return false; } } @FunctionalInterface public interface PageByIndexRangeFinder<T extends Serializable> extends Serializable { /** * Returns items within a range of indices, where indices start at 0. If the range covers indices that don't exist, * any records that are in range should be returned. No item should ever be null. For convenience of MySQL * adapters, OFFSET is startIndex and LIMIT is endIndex - startIndex + 1. * * @param startIndex Start of index range, where indices start at zero. * @param endIndex End of index range (inclusive), where indices start at zero. */ public List<T> getItems(int startIndex, int endIndex) throws Exception; /** * Returns true if finder eager loads any optional extensions of each item. This is usually false as loading * extensions results in poor performance or there are no optional extensions. */ default public boolean getLoadsDetails() { return false; } }

Updating core member fields

The ItemData class must include whether the record details need to be lazy loaded.

private static class ItemData<K,T> implements Serializable { private boolean detailsReload; // Whether further data must be added to base record to display in a detail page private K id; // Placeholder id or primary of item private T item; // Record, which is null if not yet loaded private boolean pageExtensionReload; // Whether further data must be added to base record to display in a search results page private boolean reload; // Whether item must be loaded or reloaded. public ItemData() { detailsReload = true; id = null; item = null; pageExtensionReload = false; reload = false; } public boolean getDetailsReload() { return detailsReload; } public void setDetailsReload(boolean value) { detailsReload = value; } public K getId() { return id; } public void setId(K value) { id = value; } public T getItem() { return item; } public void setItem(T value) { item = value; } public boolean getPageExtensionReload() { return pageExtensionReload; } public void setPageExtensionReload(boolean pageExtensionReload) { this.pageExtensionReload = pageExtensionReload; } public boolean getReload() { return reload; } public void setReload(boolean reload) { this.reload = reload; } }

As not all lists have details, a flag can skip the check for details. The member variables become the following

public class ListCache<K extends Serializable,T extends Serializable> implements Serializable { private ListFinder<T> baseRecordListFinder; // Strategy to load base record list in base record list pagination mode private boolean hasDetails; // Whether records have additional data for a detail page. private IdListFinder<K> idListFinder; // Strategy to load ids of list for page by ids pagination mode private SingleItemFinder<K,T> itemFinder; // Reloads single item with details private KeyExtractor<K,T> keyExtractor; private List<ItemData<K,T>> list; // List of records or placeholders private ListFinder<T> listFinder; // Strategy to load entire list in full list pagination mode private ListSizeFinder listSizeFinder; // Strategy to get list size for page by index pagination mode private PageByIdsFinder<K,T> pageByIdsFinder; // Strategy to retrieve page in page by ids pagination mode private PageByIndexRangeFinder<T> pageByIndexRangeFinder; // Strategy to retrieve page by indices private PageExtensionAssembler<T> pageExtensionAssembler; // Adds additional data for current page for base record list pagination mode private int pageSize; private PaginationMode paginationMode; private boolean reload; // Whether list must be reloaded private int selectedIndex; // Index of currently selected record, 0 based, or -1 for no selected record

Updating core functions

The getSelected function retrieves the item for the detail page and must lazy load details as needed.

/** * Checks if details need to be loaded for records, loading them if not. */ private void checkReloadItemDetail(int startIndex, int endIndex) throws Exception { Set<Integer> indicesToLoad; boolean reloadNeeded, reloadList; indicesToLoad = new HashSet<>(); for (int index = startIndex; index <= endIndex; index++) { reloadNeeded = getItemReloadNeeded(index); if (reloadNeeded) { indicesToLoad.add(index); } } if (indicesToLoad.size() > 0) { if (hasDetails && itemFinder == null) { throw new IllegalStateException("Single item finder for " + name + " list is not set but details need to be reloaded."); } else if (itemFinder == null) { reloadList = true; } else { reloadList = !loadItemDetails(indicesToLoad); } if (reloadList) { loadList(); checkSelectedIndex(); } } } /** * Determine if selected item or its extensions needs to be reloaded. */ private boolean getItemReloadNeeded(int index) { ItemData<K,T> itemData; boolean reloadNeeded; if (index >= 0) { itemData = list.get(index); if (itemData.getItem() == null || itemData.getReload()) { reloadNeeded = true; } else if (hasDetails && itemData.getDetailsReload()) { reloadNeeded = true; } else { reloadNeeded = false; } } else { reloadNeeded = false; } return reloadNeeded; } /** * Loads individual items using the item finder and returns false if no record was found for any key. */ private boolean loadItemDetails(Collection<Integer> indices) throws Exception { ItemData<K,T> itemData; T item; boolean failed; failed = false; for (int index: indices) { itemData = list.get(index); item = itemFinder.getItem(itemData.getId()); if (item != null) { itemData.setItem(item); itemData.setReload(false); itemData.setDetailsReload(false); } else { failed = true; } } return !failed; } /** * Returns selected item from list. */ public synchronized T getSelected() throws Exception { checkReloadState(); if (selectedIndex >= 0) { checkPageListData(selectedIndex,selectedIndex); checkReloadItemDetail(selectedIndex,selectedIndex); return list.get(selectedIndex).getItem(); } else { return null; } }

The getList and getPage functions must also check if items were marked for reload.

/** * Checks if items were marked for reload, loading if so. */ private void checkReloadItems(int startIndex, int endIndex) throws Exception { ItemData<K,T> itemData; Set<Integer> indicesToLoad; boolean reloadList; indicesToLoad = new HashSet<>(); for (int index = startIndex; index <= endIndex; index++) { itemData = list.get(index); if (itemData.getReload()) { indicesToLoad.add(index); } } if (indicesToLoad.size() > 0) { if (itemFinder == null) { reloadList = true; } else { reloadList = !loadItemDetails(indicesToLoad); } if (reloadList) { loadList(); checkSelectedIndex(); } } } public synchronized List<T> getPage() throws Exception { int pageNo, startIndex, endIndex; checkReloadState(); if (list.size() > 0) { pageNo = getPageNo(); pageNo = (pageNo < 1)?1:pageNo; startIndex = (pageNo - 1) * pageSize; endIndex = startIndex + pageSize - 1; endIndex = (endIndex >= list.size())?list.size() - 1:endIndex; checkPageListData(startIndex, endIndex); checkReloadItems(startIndex, endIndex); return getListUnchecked().subList(startIndex, endIndex + 1); } else { return new ArrayList<T>(); } } public synchronized List<T> getList() throws Exception { checkReloadState(); checkEntireListData(); checkReloadItems(0, list.size()-1); return getListUnchecked(); }

Functions using the list and page finders must set the detail flag for records they load.

private void checkEntireListDataByIds() throws Exception { ItemData<K,T> itemData; K id; T item; Collection<K> ids; Collection<T> loadingItems; List<Integer> removalIndices; Map<K,Integer> loadingIds; int i, index, size; boolean detailLoaded; size = list.size(); // Find items that need to be loaded ids = new ArrayList<K>(); loadingIds = new HashMap<K,Integer>(); for (i = 0; i < size; i++) { itemData = list.get(i); if (itemData.getReload()) { ids.add(itemData.getId()); loadingIds.put(itemData.getId(), i); } } // Load items from page-by-id finder loadingItems = pageByIdsFinder.getItems(ids); detailLoaded = pageByIdsFinder.getLoadsDetails(); for (T loadingItem: loadingItems) { id = keyExtractor.getKey(loadingItem); index = loadingIds.get(id); itemData = list.get(index); itemData.setDetailsReload(!detailLoaded); itemData.setItem(loadingItem); itemData.setPageExtensionReload(false); itemData.setReload(false); } // Delete items that page-by-id finder did not load as they no longer exist removalIndices = new ArrayList<Integer>(); for (K loadingId: loadingIds.keySet()) { index = loadingIds.get(loadingId); item = list.get(index).getItem(); if (item == null) { removalIndices.add(index); } } Collections.sort(removalIndices, Collections.reverseOrder()); for (int removeIndex: removalIndices) { list.remove(removeIndex); } } private void checkEntireListDataByIndexRange() throws Exception { ItemData<K,T> itemData; T item; List<T> loadingItemList; int i, size, rangeSize, lowestNotLoaded, highestNotLoaded; boolean detailLoaded; size = list.size(); // Find what needs to be loaded. If that's multiple ranges, one big load is done anyway as it's probably faster // than multiple loads lowestNotLoaded = -1; highestNotLoaded = -1; for (i = 0; i < size; i++) { if (list.get(i).getReload()) { if (lowestNotLoaded < 0) { lowestNotLoaded = i; } highestNotLoaded = i; } } if (lowestNotLoaded > -1) { rangeSize = highestNotLoaded - lowestNotLoaded + 1; loadingItemList = pageByIndexRangeFinder.getItems(lowestNotLoaded, highestNotLoaded); detailLoaded = pageByIndexRangeFinder.getLoadsDetails(); // If result size differs from reported size, shrink or grow list to fit if (rangeSize > loadingItemList.size()) { list.subList(lowestNotLoaded + loadingItemList.size(), size).clear(); } // If result size greater than requested range, WTF?, ignore the excess // Load items from page-by-index-range finder for (i = 0; i < loadingItemList.size(); i++) { item = loadingItemList.get(i); itemData = list.get(i + lowestNotLoaded); itemData.setDetailsReload(!detailLoaded); itemData.setId(keyExtractor.getKey(item)); itemData.setItem(item); itemData.setPageExtensionReload(false); itemData.setReload(false); } } } private void checkPageListDataByIds(int startIndex, int endIndex) throws Exception { ItemData<K,T> itemData; Collection<K> ids; Collection<T> loadingItems; List<Integer> removalIndices; Map<K,Integer> loadingIds; K id; T item; int i, index, listSize; boolean loadFailed; do { loadFailed = false; listSize = list.size(); // Find items that need to be loaded ids = new ArrayList<K>(); loadingIds = new HashMap<K,Integer>(); for (i = startIndex; i <= endIndex; i++) { if (i >= 0 && i <= listSize - 1) { itemData = list.get(i); if (itemData.getReload()) { ids.add(itemData.getId()); loadingIds.put(itemData.getId(), i); } } } if (ids.size() > 0) { // Load items from multi item finder loadingItems = pageByIdsFinder.getItems(ids); for (T loadingItem: loadingItems) { id = keyExtractor.getKey(loadingItem); index = loadingIds.get(id); itemData.setDetailsReload(!pageByIdsFinder.getLoadsDetails()); itemData.setItem(loadingItem); itemData.setReload(false); itemData.setPageExtensionReload(false); } // Delete items that multi item finder did not load as they no longer exist removalIndices = new ArrayList<Integer>(); for (K loadingId: loadingIds.keySet()) { index = loadingIds.get(loadingId); item = list.get(index).getItem(); if (item == null) { removalIndices.add(index); loadFailed = true; } } Collections.sort(removalIndices, Collections.reverseOrder()); for (int removalIndex: removalIndices) { list.remove(removalIndex); } } } // If there any failed items, which are removed, higher entries may // shift to take their place in the requested page and some may be unloaded. while (loadFailed); } private void checkPageListDataByIndexRange(int startIndex, int endIndex) throws Exception { ItemData<K,T> itemData; T item; List<T> loadingItemList; int i, listSize, rangeSize, lowestNotLoaded, highestNotLoaded; boolean detailLoaded; listSize = list.size(); // Find what needs to be loaded. If that's multiple ranges, one big load is done anyway as it's probably faster // multiple loads lowestNotLoaded = -1; highestNotLoaded = -1; for (i = startIndex; i <= endIndex; i++) { if (list.get(i).getReload()) { if (lowestNotLoaded < 0) { lowestNotLoaded = i; } highestNotLoaded = i; } } if (lowestNotLoaded > -1) { rangeSize = highestNotLoaded - lowestNotLoaded + 1; loadingItemList = pageByIndexRangeFinder.getItems(lowestNotLoaded, highestNotLoaded); detailLoaded = pageByIndexRangeFinder.getLoadsDetails(); // If result size less than requested range, end of source list unexpectedly reached, so shrink our list if (rangeSize > loadingItemList.size()) { list.subList(startIndex + loadingItemList.size(), listSize).clear(); } // If result size greater than requested range, WTF?, ignore the excess // Load items from page-by-index-range finder for (i = 0; i < loadingItemList.size() && i < rangeSize; i++) { item = loadingItemList.get(i); itemData = list.get(i + lowestNotLoaded); itemData.setDetailsReload(!detailLoaded); itemData.setId(keyExtractor.getKey(item)); itemData.setItem(item); itemData.setPageExtensionReload(false); itemData.setReload(false); } } } private void loadList() throws Exception { List<K> idList; List<T> itemList; int listSize; switch (paginationMode) { case BASE_RECORD_LIST: itemList = baseRecordListFinder.getList(); if (itemList == null) { itemList = new ArrayList<T>(); } setList(itemList, falsebaseRecordListFinder.getLoadsDetails(), baseRecordListFinder.getLoadsDetails()); break; case FULL_LIST: itemList = listFinder.getList(); if (itemList == null) { itemList = new ArrayList<T>(); } setList(itemList, true, listFinder.getLoadsDetails()); break; case PAGE_BY_IDS: idList = idListFinder.getIds(); if (idList == null) { idList = new ArrayList<K>(); } setIdList(idList); break; case PAGE_BY_INDEX_RANGE: listSize = listSizeFinder.getSize(); setListSize(listSize); break; } } private synchronized void setList(List<T> value, boolean pageExtensionsLoaded, boolean detailLoaded) { ItemData<K,T> itemData; reload = false; list = new ArrayList<ItemData<K,T>>(value.size()); for (T item: value) { itemData = new ItemData<K,T>(); itemData.setDetailsReload(!detailLoaded); itemData.setId(keyExtractor.getKey(item)); itemData.setItem(item); itemData.setPageExtensionReload(!pageExtensionsLoaded); itemData.setReload(false); list.add(itemData); } checkSelectedIndex(); } private synchronized void setListSize(int listSize) { ItemData<K,T> itemData; int i; reload = false; list = new ArrayList<ItemData<K,T>>(listSize); for (i = 0; i < listSize; i++) { itemData = new ItemData<K,T>(); itemData.setDetailsReload(true); itemData.setId(null); itemData.setItem(null); itemData.setPageExtensionReload(false); itemData.setReload(true); list.add(itemData); } checkSelectedIndex(); } public synchronized void setBaseRecordListAndFinder(List<T> list, ListFinder<T> baseRecordListFinder) { setList(list, falsebaseRecordListFinder.getLoadsDetails(), baseRecordListFinder.getLoadsDetails()); this.baseRecordListFinder = baseRecordListFinder; this.idListFinder = null; this.listFinder = null; this.listSizeFinder = null; this.pageByIndexRangeFinder = null; this.paginationMode = PaginationMode.BASE_RECORD_LIST; setSelectedIndex(0); } public void setBaseRecordListFinder(ListFinder<T> baseRecordListFinder) { setList(new ArrayList<>(), falsebaseRecordListFinder.getLoadsDetails(), baseRecordListFinder.getLoadsDetails()); this.baseRecordListFinder = baseRecordListFinder; this.idListFinder = null; this.listFinder = null; this.listSizeFinder = null; this.pageByIndexRangeFinder = null; this.paginationMode = PaginationMode.BASE_RECORD_LIST; markReload(); } public synchronized void setListAndFinder(List<T> list, ListFinder<T> listFinder) { setList(list, true, listFinder.getLoadsDetails()); this.baseRecordListFinder = null; this.idListFinder = null; this.listFinder = listFinder; this.listSizeFinder = null; this.pageByIndexRangeFinder = null; this.paginationMode = PaginationMode.FULL_LIST; setSelectedIndex(0); } public void setListFinder(ListFinder<T> listFinder) { setList(list, true, listFinder.getLoadsDetails()); this.baseRecordListFinder = null; this.idListFinder = null; this.listFinder = listFinder; this.listSizeFinder = null; this.pageByIndexRangeFinder = null; this.paginationMode = PaginationMode.FULL_LIST; markReload(); }

Next part

Continued in Composite Cache.