I have seen an interesting issue while using HTTP Adapter and thought about sharing with you all. Many times, I saw that we posted duplicate messages to HTTP URL from our HTTP Send Port or sometimes orchestration which is sending message to this HTTP Send Port, got suspended with error “The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended.” and which typically refers to a zombie message in BizTalk world.
On analysis, we found it to be an issue with one-way HTTP Adapter while an orchestration is sending a message to it with Delivery Notification enabled. This happens when we have another subscriber (a Send Port) subscribing to the same message going to this HTTP Send Port.
Let me show you how we can easily reproduce this issue.
1. For the repro, we will use the HelloWorld SDK sample solution. First, lets open the HelloWorld.sln Solution which it typically located under \SDK\Samples\Orchestrations\HelloWorld folder. On my box, the directory is C:\Program Files (x86)\Microsoft BizTalk Server 2013\SDK\Samples\Orchestrations\HelloWorld.
2. Open the HelloOrchestration.odx and go to the SendInvoicePort Port properties and change the Delivery Notification to Transmitted.
3. Build the solution and deploy it to any BizTalk application. You may need to sign the assembly with a strong name key file.
4. Create a one way Send Port with HTTP Adapter and point it to any HTTP URL where messages can be posted. In my case, I am using another HTTP receive location for receiving messages over HTTP.
5. Create a Receive Port with a receive location and configure it with File Adapter and XMLReceive pipeline.
6. Bind the HelloWorld orchestration with the Receive Port and HTTP Send Port created in earlier steps.
7. Now the HelloWorld sample is good to test after you have recycled the host instances and started all the artifacts used. If you drop the SamplePOInput.xml (located under the HelloWorld folder) in the file location configured earlier, the orchestration will post the message to HTTP URL through HTTP Send Port. If everything is set up correctly, you will see an Invoice xml message posted to the HTTP URL.
8. Till here, everything is working as expected. But before we repro the issue, let me show you how Delivery Notification is working here.
For this, we will stop the HTTP Send Port and drop one more input file.
Now query for All In-Progress Service Instances and you will see that the HTTP Send Port is in suspended (resumable) state and Orchestration is in Dehydrated stage. Send Port is suspended as we have kept it stopped and orchestration is dehydrated as it waiting for the the ACK from the Send Port since we have enabled the Delivery Notification.
If you query for the subscriptions, you will find that there is an instance subscription for our orchestration instance which is in dehydrated state, and it has got a subscription on Correlation Token as shown below.
Now, if you open the suspended Send Port instance, and go to the context of the message, you will see Correlation Token in the context property and the value matches to the value found above in the subscription of the orchestration instance. Also note that there is another property AckRequired with value True. This tells the Adapter to publish an ACK back, with the same correlation token, after it is done processing so that orchestration can come to know about it. That is how the Delivery Notification works.
Now this explained, let’s go back to repro our issue.
9. We will create another one way Send Port with File Adapter and point it to any file location. Add the below filter on this Send Port so that it subscribes to the message going to the HTTP Send Port directly from the MessageBox.
BTS.MessageType == http://HelloWorld.InvoiceSchema#Invoice
This will also mean that both the HTTP and File Send Port will receive the same message. HTTP Port is directly bound to the Orchestration and File Port will subscribe to the same message directly from the MessageBox.
10. Keep both the Send Port’s (i.e. both File and HTTP) in the stopped state and drop the input file. Now you will see that both the Send Ports are in stopped
Now, point to note is that both have received the same copy of message and if you open the context for both the suspended messages, they will have the same Correlation Token in the Context and will also have the AckRequired as True. So both the Send Ports are capable to sending the Ack back to the orchestration instance after they got processed and orchestration cannot differentiate from where the Ack is coming as it is only cares about the right value of Correlation Token.
11. Right click the suspended File Send Port message and resume it. You will find that both File Send Port message and orchestration have completed. But HTTP Send Port message is still suspended as we have not resumed it.
12. Now resume the suspended HTTP Send Port message, and after a while you will observe that Send Port is stuck in active state and there is also a Routing Failure Report. 
If you check at the receive HTTP URL, you will observe that the URL received the message successfully, but the message at HTTP Send Port is stuck in Active state for ever.
So what is happening is that after the HTTP Send Port has finished processing (i.e. posting to HTTP URL), it is trying to publish the ACK to the MessageBox but since now the orchestration has already completed, it is not able to find any subscriber for it and failing with the Routing Failure Report. But somehow I believe that adapter is not handling it properly and getting stuck with active state.
13. So now if I go and just recycle the host instance, you will observe that a new Routing Failure Message is suspended and Send Port is still in Active state. And if you check at the receiving HTTP URL, another copy of message would have been received. So that means, every time we recycle the host instance, a duplicate message will be sent to the HTTP URL.
Note that the above we tried to reproduce a scenario, where the file adapter got processed first and sent the ACK back to the Orchestration first while HTTP Adapter took some time to get processed.
Ideally, the HTTP Send Port should not be stuck in Active state and it should have been suspended saying no subscribers found for the ACK. If we use a File adapter instead of HTTP Adapter and try to repro it, we will see that it will get suspended with error “The published message could not be routed because no subscribers were found. “.
This scenario can easily occur in Prod, if someone create a Send Port (maybe for archival or something else) with filter on MessageType for the message going to HTTP Send Port.
We can also easily reproduce zombie with the same repro. Keep both the Send Ports in suspended state and drop the input file again. Now resume both the suspended Send Ports together and if you got the right timing (i.e. both adapters were able to publish their ACK before the orchestration completes), you will see that the orchestration got suspended with error “The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended.”. It means that orchestration received the Ack which got published first and the orchestration started processing further. And the second Ack, which got published after the first but before the orchestration completes, got treated as a zombie message.
With this, we have reproduced both the issues. The first issue, i.e. HTTP Send Port stuck in active and sending duplicates at every recycle, is more like an issue with HTTP Adapter as I am not able to reproduce it with any adapter. But the second issue, i.e. about orchestration getting suspended is a more legitimate issue and can occur with any adapter. To resolve the second issue, we can suppress the AckRequired by using a pipeline at the second Send Port (in our repro it will be File Send Port). Below is a snippet of the code, we can use at the pipeline at the Send Port which will make the AckRequired property to false, so that the adapter will not publish the Ack to the MessageBox.
if (inmsg.Context.Read("AckRequired", "http://schemas.microsoft.com/BizTalk/2003/system-properties") != null)
{
if (inmsg.Context.Read("AckRequired", "http://schemas.microsoft.com/BizTalk/2003/system-properties").ToString() == "True")
{
inmsg.Context.Write("AckRequired", "http://schemas.microsoft.com/BizTalk/2003/system-properties", "False");
}
}
Note that I have reproduced this issue on BizTalk 2009 and as well as BizTalk Server 2013 R2.
That’s all for now. Let me know what you think about this post.
It means, that BizTalk internally has create a subscription for the Send Port although we have not explicitly defined any filters for it.
Atin Agarwal's Integration Blog 