LibHistoire - Guild History

sirinsidiator 100 阅读 172 评论 265 点赞

What is this library for?
Up until now, addons had to send guild history requests to the server individually, which in the past has lead to huge performance problems as more and more players started using these addons and ultimately forced ZOS to throttle how fast they are allowed to send automated requests for the guild history. In order to alleviate this problem and take a lot of the complexity away from individual addons, this library was born. Instead of having each addon send requests, it will all be taken care of by LibHistoire and they will simply become passive listeners.

The library will take care of loading missing history from the server in the background and store everything locally so it never has to request the same time range more than once.

Right now the game provides the history in an inverse order (future to past) of what addons actually try to use (past to future), so before the library passes any events to addons, it will attempt to load the full time range since you last played the game. During this time, it will keep them in a temporary "unlinked" state and only when it encounters the last stored event, it will save them and start sending events to listening addons. If you quit the game before that happens, all progress will be lost and it will have to start over from the beginning the next time you log in.

You can speed this process up by manually requesting missing history via the guild history menu, as this action won't be subject to the cooldown any addon has to deal with. Another option is to tell the library to force a link with what data it has got and create a hole in your history. You can always manually request more history events and trigger a rescan which will find any missing entries later.

Unlike other addons, the library won't request the full range of history provided by the server when you first use it. Instead it will immediately force a link and start collecting data from there. If you want to collect older history in that situation, you will need to manually request it and then hit the rescan button for that guild and category.

Dependencies
The following dependencies are required by LibHistoire:

LibAsync - to minimize fps loss while processing history events
LibCustomMenu - for the options menu of the status window
LibDebugLogger - for logging useful debug information in case something goes wrong

Some observations on what the server does

Requesting older history events via an addon starts a cooldown
The cooldown is separate for each guild and category, meaning an addon can send up to 20 requests at once
Simply put the interval between requests depends on how many players are currently online and is dynamically controlled by the server
Interval can be anything from a few seconds (on PTS) to several minutes during primetime EU. I've often seen values between 3 and 4 minutes while developing the library
New events are pushed automatically by the server with no way for addons to influence that process
The time between new events varies greatly and seems to depend on the how many players are online and how many events a guild produces
Sometimes the updates arrive almost instantly, sometimes they don't arrive for several hours - I've had to wait 2-3 hours quite often before any new guild history arrived
Ocassionally the updates never arrive (even after waiting for 5 hours) and restarting the game is the only way to force an update in that situation

User Interface

The status icon on the bottom right of the guild history symbolizes the link status of the currently viewed category in the selected guild. On hover it will show a tooltip that gives information about the stored history and unlinked events.

The new guild history status panel will provide an overview of what LibHistoire has stored and what is missing.

On the left side it will show each guild and the overall progress, on the right side it will show the categories for the currently selected guild.
Clicking on a guild or category will update the selection in both the status window and the guild history menu accordingly.
When you hover over any of the entries, a tooltip will show you the same information as the tooltip in the guild history menu.
The category entries also house the force link button (while the category is unlinked) and the rescan button (after it has linked up with stored events).
On the bottom of the panel you can see an icon which symbolizes the overall state and gives some general information about what is happening when you hover over it.
The cog button on the top right will open a context menu with an option to unlock the window so it can be moved and an option to hide it (same as the button on the bottom left of the guild history)

Special Thanks
FooWasHere who helped me test how the history behaves on rank and permission changes
ZOSDanBatson for answering my many questions about the history API
Everyone else who helped me test this and gave me feedback

For Developers
Why should you use it?

It minimizes the amount of requests sent by addons to the absolute minimum and if every addon starts using it, it will likely lower the cooldown the server chooses, so everyone gets their data faster.
It takes care of all the complexity that comes with requesting the history. There are many special cases you probably didn't even think about. The lib will handle them all for you.
It stores the history locally, so an addon can access time ranges that would be impossible via the game api
The data format minimizes impact on load time and disk space usage (compared to other addons like MM or ATT)

How does it work
The library listens to all incoming data and determines for each individual event if it has already been stored locally. It also takes care of sorting the events in the correct historic order and waits until the hole since the last login has been filled before saving them to disk and passing them to listeners.
When a listener starts, it will first iterate over available stored events, then wait for "unlinked" events to link before it iterates over those and finally start passing along newer events whenever they arrive. This is all done via LibAsync, so you will only get as many events per frame as you can safely process without affecting performance.
To avoid having to deserialize all stored events, it offers ways to select a starting point either by specifying an eventId or a timestamp*. It will do a binary search for the closest event and start from there instead of the beginning of the stored history.

*timestamps may not be 100% accurate as event times are returned from the game api as seconds since an event and manually converted to an absolute timestamp. This brings a certain inaccuracy with it and can lead to events being stored with timestamps that do not reflect their actual order. If precision is required it may be best to specify a timestamp a few seconds before the actual target time and use eventIds to identify an actual border for a time range.

If you find a problem, feel free to open an issue over on github, or leave a comment here on ESOUI.

API Reference
Callbacks
Can be accessed via the LibHistoire.callback table.

Code:

INITIALIZED()

Fired when Histy has finished initializing and is ready. Any call to the api (aside of RegisterCallback) should be done after it has fired.

Code:

HISTORY_RESCAN_STARTED(integer guildId, integer category)

Fired when the user or library triggered a rescan of the history.

Code:

HISTORY_RESCAN_ENDED(integer guildId, integer category, integer numEventsBefore, integer numEventsInside, integer numEventsAfter, boolean foundInvalidEvents)

Fired after a category rescan has finished. Invalid events are product of a bug where the game won't set the correct event time for a few seconds and the library will ignore them.

Library API

Code:

RegisterCallback(LibHistoire.callback type, function callback)

Code:

UnregisterCallback(LibHistoire.callback type, function callback)

Unregister from a callback. Usage is the same as with CALLBACK_MANAGER:UnregisterCallback.

Code:

LibHistoire.GuildHistoryEventListener listener = CreateGuildHistoryListener(integer guildId, integer category)

Creates a GuildHistoryEventListener object which can be configured before it starts listening to history events, or nil if guildId or category are not valid.
WARNING: Make sure you only call it after the INITIALIZED callback has fired, otherwise it may return nil.

GuildHistoryEventListener

Code:

string key = GetKey()

Returns a key consisting of server, guild id and history category, which can be used to store the last received eventId.

Code:

integer guildId = GetGuildId()

Returns the guild id of the listener.

Code:

integer category = GetCategory()

Returns the category of the listener.

Code:

integer eventCount, number processingSpeed, number timeLeft = GetPendingEventMetrics()

Returns information about history events that need to be sent to the listener.

eventCount - the amount of stored or unlinked events that are currently waiting to be processed by the listener
processingSpeed - the average processing speed in events per second or -1 if not enough data is yet available
timeLeft - the estimated time in seconds it takes to process the remaining events or -1 if no estimate is possible

Code:

boolean success = SetAfterEventId(id64 eventId)

Accepts an eventId (id64). The nextEventCallback will only return events which have a higher eventId when it is set. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = SetAfterEventTime(integer eventTime)

If no afterEventId has been specified, the nextEventCallback will only receive events after the specified timestamp. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = SetBeforeEventId(id64 eventId)

Accepts an eventId (id64). The nextEventCallback will only return events which have a lower or equal eventId when it is set. Once the specified eventId has been passed, the listener will automatically stop and call the function specified with SetIterationCompletedCallback. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = SetBeforeEventTime(integer eventTime)

If no beforeEventId has been specified, the nextEventCallback will only receive events before (including) the specified timestamp. Like with beforeEventId, the listener will stop and call the iteration completed function when the specified eventTime has been reached. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = SetIterationCompletedCallback(function callback)

When an end criteria is set, this callback will fire when the listener has stopped automatically.

Code:

boolean success = SetStopOnLastEvent(boolean shouldStop)

When set to true, the iteration will stop on the last stored event and fire the iterationCompleted callback instead of waiting for new events to arrive.

Code:

boolean success = SetTimeFrame(integer startTime, integer endTime)

A convenience method to specify a time range which includes the startTime and excludes the endTime. See SetAfterEventTime and SetBeforeEventTime.

Code:

boolean success = SetNextEventCallback(function callback)

Sets a callback which is passed stored and received events in the correct historic order (sorted by eventId). Will return true if it was called while the listener is not running, or false otherwise.
The callback has the following signature:

Code:

function(GuildEventType eventType, id64 eventId, integer eventTime, variable param1, variable param2, variable param3, variable param4, variable param5, variable param6)

More details about param1-6 can be found in guildhistory_shared.lua in the UI source code.

Code:

boolean success = SetMissedEventCallback(function callback)

Sets a callback which is passed missed events that are discovered during a rescan, ignoring any afterEventId or afterEventTime values. See SetNextEventCallback for details on the callback. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = SetEventCallback(function callback)

A convenience function which will set both callback types in one go. See SetNextEventCallback for details on the callback. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = Start()

Signals that the listener can start sending data to the addon. Will return true if it was called while the listener is not running, or false otherwise.

Code:

boolean success = Stop()

Signals that the listener to stop sending data to the addon. Will return true if it was called while the listener is running and has been stopped, or false otherwise.

Code:

boolean success = IsRunning()

Returns the current state of the listener.

Examples
Iterate over all stored events

Lua Code:


LibHistoire:RegisterCallback(LibHistoire.callback.INITIALIZED, function()

    local listener = LibHistoire:CreateGuildHistoryListener(guildId, category)

    listener:SetEventCallback(function(eventType, eventId, eventTime, param1, param2, param3, param4, param5, param6)

        -- do something with the event data

    end)

    listener:Start()


end)

This will simply iterate over all stored events and then listen to any newly received data, including anything that will be found on a rescan.

Retrieve a time range

Lua Code:


LibHistoire:RegisterCallback(LibHistoire.callback.INITIALIZED, function()

    local listener = LibHistoire:CreateGuildHistoryListener(guildId, category)

    listener:SetTimeFrame(startTime, endTime)

    listener:SetNextEventCallback(function(eventType, eventId, eventTime, param1, param2, param3, param4, param5, param6)

         -- do something with the event data

    end)

    listener:Start()


end)

This example shows how to iterate over a specific time frame and then stop. SetTimeFrame is preferable over just checking the eventTime in the callback, as it will be faster than just iterating everything.

Retrieve sales for different guilds and continue from last event

Lua Code:


LibHistoire:RegisterCallback(LibHistoire.callback.INITIALIZED, function()

    local function SetUpListener(guildId, category)

        local listener = LibHistoire:CreateGuildHistoryListener(guildId, category)

        local key = listener:GetKey()

        listener:SetAfterEventId(StringToId64(saveData.lastEventId[key]))

 
        listener:SetNextEventCallback(function(eventType, eventId, eventTime, param1, param2, param3, param4, param5, param6)

            -- the events received by this callback are in the correct historic order

            saveData.lastEventId[key] = Id64ToString(eventId)

        end)

 
        listener:SetMissedEventCallback(function(eventType, eventId, eventTime, param1, param2, param3, param4, param5, param6)

            -- events in this callback are out of order compared to what has been received by the next event callback and can even have an eventId smaller than what has been specified via SetAfterEventId.

        end)

        listener:Start()

    end

 
    for i = 1, GetNumGuilds() do

        SetUpListener(GetGuildId(i), category)

    end


end)

This example will register listeners for all guilds and keep track of the last event id that has been processed by the addon. That way the library will pass every event to the listener only once. It will also handle events that have been found later during a manual rescan.