You need to 'end' the tabset. You can do this by going up a heading level and using {-}.
Here's an example, using essentially what you've put together. (I used the output html_document with no additional options.)
## Analysis {.tabset .tabset-fade .tabset-pills}
### 1
### 2
### 3
## {-}
### A {.tabset .tabset-fade .tabset-pills}
#### 3.1
A graph code here to be shown
```{r p1, echo=FALSE}
plot(x = 1, y = 2)
```
#### 3.2
A graph code here to be shown
```{r p2, echo=FALSE}
plot(x = 4, y = 10)
```
## {-}
---
### B {.tabset .tabset-fade .tabset-pills}
#### 3.3
A graph code here to be shown
```{r p3, echo=FALSE}
plot(x = 1, y = 10)
```
#### 3.4
A graph code here to be shown
```{r p4, echo=FALSE}
plot(x = 1:100, y = 100:1)
```
In the image, I made it narrow so it wouldn't be to large for SO.

Update starts here
I'm not entirely sure if I understand your comment, so you'll have to let me know if this clarifies things or if this still isn't what you were looking for, let me know in more detail what you are expecting.
I see that when I created A and B it was at the same heading level as 1, 2, and 3. I'm guessing that this is where the problem lies from your perspective.
I opted to put together a much more robust example of nested tab sets. Because this is all nested, there are no 'stops' needed, either. (BTW, I added number-letter combinations so that you could clearly see that each view presents different content.)
## Analysis {.tabset .tabset-fade .tabset-pills}
### 1
#### 1 A {.tabset .tabset-fade .tabset-pills}
##### 1.1
```{r p11, echo=FALSE}
plot(x = 1, y = 2)
```
##### 1.2
```{r p12, echo=FALSE}
plot(x = 1, y = 2)
```
### 2
#### 2 A {.tabset .tabset-fade .tabset-pills}
##### 2.1
```{r p21, echo=FALSE}
plot(x = 1, y = 2)
```
##### 2.2
```{r p22, echo=FALSE}
plot(x = 1, y = 2)
```
### 3
#### 3 A {.tabset .tabset-fade .tabset-pills}
##### 3.1
```{r p1, echo=FALSE}
plot(x = 1, y = 2)
```
##### 3.2
```{r p2, echo=FALSE}
plot(x = 4, y = 10)
```
----
#### 3 B {.tabset .tabset-fade .tabset-pills}
##### 3.3
```{r p3, echo=FALSE}
plot(x = 1, y = 10)
```
##### 3.4
```{r p4, echo=FALSE}
plot(x = 1:100, y = 100:1)
```
