Sitecore Event Queue Visual Diagram

February 27, 2018

Blog | Technology | Sitecore Event Queue Visual Diagram
Sitecore Event Queue Visual Diagram

The Sitecore Event Queue is a powerful piece of technology. In my experience, it is often misunderstood. To help clear up some common misconceptions for beginners, and possibly to teach a thing or two to experienced Sitecore developers, I’ve prepared an animation illustrating an example use-case of the Event Queue. The visual diagram cuts a few corners to simplify the process a bit. These will be addressed later in this post but focuses on the main operation and execution of the Event Queue.

System Event Queue

First off- every database contains an [EventQueue] table with the same schema, so it’s not entirely correct to simply say “the event queue”- though this is uttered quite often in Sitecore Slack or Sitecore Stack Exchange. However, when developers talk about “the event queue” they are almost always talking about the System Event Queue. The System Event Queue is the database defined in the Sitecore config under /sitecore/eventing/providers/sitecore[systemDatabaseName]. By default, this is the Core database. The systemDatabase is special for a few reasons:

  • Sitecore makes the assumption that all web servers in a cluster can connect to this database
  • Sitecore assumes all web servers in a cluster have the same database set as the systemDatabase
  • Queuing a RemoteEvent via EventManager uses the systemDatabase
    • Sitecore.Eventing.EventManager.QueueEvent(TEvent event);

Note: It is possible to queue an event directly in a particular database via (DatabaseObject).RemoteEvents.Queue.QueueEvent(TEvent event);

Event queues are essentially messaging systems for your web servers. Any web servers in a cluster (CM, CD, Processing, Reporting, etc.) that share a common database, can issue instructions to be executed on a remote machine.

Other Common Areas of Confusion

  • When queuing an event, the event does not execute on the remote servers immediately.
  • It is worth noting that when Sitecore checks “the event queue” every 2 seconds (default value) it checks the [EventQueue] of all Sitecore databases, not just the systemDatabase.
  • Any instance of the word “events” in this post or pertaining to the EventQueue refers to a RemoteEvent. This is not to be confused with a local event, such as <publish:end:remote>. The best guide on the internet showcasing the difference between Remote Events and Local Events can be found here.
  • A server must understand a RemoteEvent in order to process it correctly. When an app pool on a web server starts, all RemoteEvents are stored in a SubscriptionMap. This mapping essentially tells Sitecore “If the following type is found in the EventQueue, execute this method with the data provided.”

Diagram

Imagine a small web cluster- 1 Content Management server (CM), 2 Content Delivery servers (CD1/CD2). All web servers have their systemDatabaseName set to the Core database. Pay attention to the highlighted sections. To get a more detailed explanation of each step, please compare the number on the slide to the number listed below.

  1. Nothing
    1. This is the start of the animation, but keep in mind this process is constant when the app pools are running. If an app pool goes down, as soon as it restarts, this process continues where it left off.
  2. EventQueue Configured on App Start
    1. On each web server, Sitecore reads the EventProvider section and initiates an AlarmClock instance. This is a Sitecore class that runs a method after a certain time has expired – a Heartbeat. It runs continuously when the app pool is running.
      1. In this case, the AlarmClock event has a Heartbeat that executes the method EventProvider.RaiseQueuedEvents every 2 seconds.
      2. RaiseQueuedEvents checks all Sitecore databases for new Events to process.
  3. Web servers register their presence
    1. This is the first instance of artistic liberty. The EQSTAMP is slightly more sophisticated than what is presented, however, the premise is still true.
      1. All web servers connecting to the event queue write a record to the [Properties] table.
      2. The [Key] column identifies the instance of the web server, prefixed with “EQSTAMP” (Event Queue Stamp)
      3. The [Value] column identifies the TimeStampvalue in the [EventQueue] table of the event that was last processed. The data types between these two fields is not a 1 to 1 match as illustrated in the diagram, but for the illustration purposes, the diagram is sufficient.
  4. Scheduled polling
    1. Every 2 seconds while the app pools are running, each web server queries the event queue of all databases it is connected to for new events. The query, in basic terms, is as follows:
      1. Find all events greater than my EQSTAMP where (the InstanceName is equal to my InstanceName and the event is set to RaiseLocally OR the InstanceName is not equal to my InstanceName and the event is set to RaiseGlobally)
  5. RebuildIndex event queued
    1. The CM server queued a RebuildIndex event and set it to run globally, but not locally.
    2. All servers are still continuously polling the EventQueue’s at this time.
  6. CD servers see the RebuildIndex event
    1. The CD servers register the event because their InstanceName does not equal the InstanceName in the event (CM) and it is set to RaiseGlobally.
  7. CD servers Raise the RebuildIndex event
    1. The CD servers independently execute the code associated with this event.
  8. CD servers update last processed event
    1. Once the CD servers have completed execution, their EQSTAMP is updated in the [Properties] table indicating that they’ve processed this event.
  9. Scheduled polling
    1. Normal operation. All web servers continue to poll the Event Queue.
  10. Publish event queued
    1. A content author queued a publish from within Content Editor. This queues a Publish event in the System EventQueue with RaiseLocally and RaiseGlobally set to true.
  11. All servers see the Publish event
    1. Any event with RaiseLocally and RaiseGlobally set to true will be pulled by all web servers.
  12. All servers Raise the Publish event
    1. The publish only occurs on a single server. There is built-in logic in the publishing process to help determine which server performs the publish (i.e. executes the SQL queries that move the items from master to web). For a more in-depth explanation of why RaiseLocally exists as an option, view my related post.
  13. Publish initiates on CM server
    1. In the example setup (CM + 2 CDs) the publish occurs on the CM server.
  14. All servers update last processed event
    1. Because all servers registered the event, they all update their EQSTAMP to indicate they’ve processed the event.

Did You Learn Something?

The Sitecore Event Queue has a lot of moving parts. Even with this detailed post, it’s really only scratching the surface. It is easy to spend hours, even days combing through the implementation (ask me, I would know). Like anything else Sitecore-related, the best way to learn about it is to decompile the source and step through the code. With a proper, base understanding, it can be leveraged to good use in the same way pipelines, agents, or any of the more common extension points.

Feel free to reach out to me on Sitecore Slack (jraps) or Twitter if you have any questions.

John Rappel

Technical Lead
Tags
  • event queue
  • remote events

Recent Work

Check out what else we've been working on